Camera distortions are mainly divided into two types: radial distortion and tangential distortion.

(1)Radial distortion (pincushion and barrel distortion): Light rays bend more sharply farther from the lens center than closer to the center.

The figure above shows the two types of radial distortion.

Tangential distortion: The lens is not perfectly parallel to the image plane—that is, the angle between the sensor and the lens during assembly is inaccurate.

This article mainly discusses the radial distortion model and the depth camera distortion model.

· Radial distortion correction

Many resources introduce the radial distortion model, and most of them jump right into the model’s mathematical formula:

To those who don’t understand, this will surely be utterly baffling—why is the distortion model structured this way rather than another? In this article, I’ll introduce the specifics of the radial distortion model in a way that reflects my own understanding.

In the above formulas, uˉ,vˉ denote the pixel coordinates of a point on the distorted image; u,v represent the pixel coordinates of the corresponding point on the corrected image. u0,v0 are the pixel coordinates of the center point of the corrected image, while u1,v1 are the pixel coordinates of the center point of the distorted image. x,y stand for the offset pixel counts of a point on the corrected image along the horizontal and vertical axes relative to the image center, i.e., the normalized image coordinates. Therefore, x2+y2 can be interpreted as the squared radius of a circle centered at the image center.

These two formulas calculate the pixel deviation compensation for the horizontal and vertical directions of the image respectively, with the compensation coefficient expressed as: [k1(x2+y2)+k2(x2+y2)2] It can be observed that the distribution of the compensation coefficient follows a quadratic function. Higher-order terms k3 and k4 can also be appended to the compensation coefficient, yet the first two terms are sufficient for practical engineering applications.

My understanding of this distortion model is that points at the same radius from the image center have the same compensation amount, while points at different radii follow a quadratic function. As shown in the figure, points on the same-radius circle have identical compensation amounts.

Therefore, the distortion model can be expressed by the formula shown above. Here, let’s make a simple extension: if the distortion distribution is not a circle but rather another shape—for example, an ellipse—So, the compensation coefficient.

The coordinates x and y also follow an elliptical distribution. I haven’t carefully derived and verified the specific form yet.

· Depth Camera Distortion Model

There are two types of distortion in depth cameras. The first type is the same kind of distortion found in RGB cameras—just as described above. This type of distortion affects the two-dimensional plane and can be corrected using a checkerboard pattern. To clearly observe and detect the corners of the checkerboard, an infrared laser image captured by the depth camera is used. The infrared laser image is paired with the depth map; for example, the infrared image and depth map captured by Microsoft’s Kinect v2 camera are matched in this way. The second type of distortion is depth distortion. If a depth camera is used to capture an image on a flat surface, theoretically, every pixel in the depth image would have the same value, since the captured image represents a planar surface and the pixel values in the depth map directly correspond to depth measurements. However, in practice, the depth maps obtained are not perfectly smooth. The edges of the depth map tend to exhibit significant distortion, resulting in point clouds that look like those shown in the figure below.

It’s shaped like a bowl.

· Depth camera calibration

· Direct Compensation Method

This correction method first assumes that the pixel at the center of the depth camera is free of distortion. The pixel value at the camera’s center is then taken as the reference value for distortion correction. For pixels at other locations in the depth map (specifically, the median values of surrounding pixels), the difference between their values and the reference value is calculated and stored in a lookup table. When correcting other images, the values from this table are used as compensation factors. This method requires that, during correction, the distance from the camera to the calibration plane be the same as or close to the camera’s actual working distance.

· Camera correction (adjustment)

x,y,z represent camera coordinates, where x′ and y′ are normalized coordinates with z=1, meaning the depth distance equals 1.

x′, y′ are coordinates under ideal conditions, while x′′, y′′ stand for distorted coordinates. Hence, lens distortion occurs at this stage.

Distortion correction refers to solving for x′, y′ given the distorted coordinates x′′, y′′.