Research on non-linear visual matching model under inherent constraints of images

The environment of visual imaging process consists of scene illumination, light propagation, illumination of image plane and signal processing. Natural scenery is ever-changing with different shapes. It is necessary to know the background characteristics of various radiation sources and targets, whether the scenery can be imaged and with what means it is suitable to be converted into an image. First, most of the images used in image processing are collected according to the principle of reflected light imaging. In the design of actual imaging system, it is generally simplified to Lambert model, and the ideal diffuse reflector is the typical Lambert [10], where the scenery emits light in all directions. Take a point in the scene as an example to analyse the light propagation. Figure 1 is a sectional view of an imaging system, and point P emits spherical waves; when it reaches the lens system, due to the differences between points on the surface of spherical waves and the refractive index of the lens, the light rays converge to point P′ of the image plane. Due to the coaxial sphericity of the imaging system, the object point P corresponds to the only image point P′. When the spherical waves emitted by all points in the field of view can be received by the imaging system, a two-dimensional image of the scene is formed.

Fig. 1

As shown in Figure 2, the light received by the human eye reaches the fundus, and passes through the human eye imaging and the eyepiece after reflection, thereby forming an image at plane S behind the eyepiece. Then, the light passes through the reducing glass, and the common objective glass receives light rays where the plane S becomes the common focal plane of the eyepiece and the two lenses behind it. Afterwards, the imaging beam forms two parallel beams after sharing the objective lens, and the distance between the two parallel beams is increased by the reflector group, which provides space for adding mirror of illumination light path and OCT mirror [11]. Through the focusing lens and CCD camera, the images of the two parallel lights are collected, individually. The focusing system can adjust the range of the camera-receiving surface, so as to obtain the appropriate fundus retinal images. The common objective lens and the structure in front of the objective lens of the imaging system are mainly designed by using principle of Kepler telescope system, and the parallax principle of binocular vision is used in the two-way camera shooting after sharing the objective lens.

Fig. 2

The physical world is imaged on the photo sensor through the camera model, and is stored on the storage device after analogue–digital conversion. In this process, it involves the transformation of four coordinate systems, including the world coordinate system (X_W, Y_W, Z_W), the camera coordinate system (X_C, Y_C, Z_C) and the image coordinate system (X_P, Y_P, Z_P).

2.2.1

Linear model

To transform a point P in space into a point P′ in the image, it is necessary to master the physical meaning and mathematical transformation relationship of the four coordinate systems. Without considering the distortion, that is, under the ideal perspective condition, the principle of transformation of object point into image point is shown in Figure 3. P is located in the real space, and its position relationship can be accurately expressed only by determining the reference frame in the 3D scene [12]. The coordinate system with O_W is called as the origin, and X_W, Y_W and Z_W as the axes of the free reference coordinate system, where the results obtained after the transformation of the following three coordinate systems are also based on this coordinate system. Under the reference coordinate system, the object point to the image point has to undergo the transformation of three coordinate systems, which are the camera coordinate whose origin is the optical centre of the camera; system that paralleled to the camera coordinate axis, where the image coordinate system on the Z_C axis of origin; and the pixel coordinate system in the same plane as the image coordinate system. The unit of camera coordinate system and image coordinate system is the actual length unit, and the pixel coordinate system displayed by the computer needs to be transformed by the actual length of each pixel point to obtain the digital pixel coordinate [13].

Fig. 3

Point P in space is a point of the actual object, and the difference of coordinates is only relative to different reference systems, which shows that reference is very important when determining the position of spatial points. The world coordinate system is transformed into the camera coordinate system through transformation of rotation and translation, and the mathematical expression of coordinate transformation in different reference frames is shown in Eq. (1). (1) [XCYCZC]=R[XwYwZw]+T=[R11R12R13R21R22R23R31R32R33][XwYwZw]+[TxTyTz] \left[ {\matrix{ {{X_C}} \cr {{Y_C}} \cr {{Z_C}} \cr } } \right] = R\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr } } \right] + T = \left[ {\matrix{ {{R_{11}}{R_{12}}{R_{13}}} \cr {{R_{21}}{R_{22}}{R_{23}}} \cr {{R_{31}}{R_{32}}{R_{33}}} \cr } } \right]\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr } } \right] + \left[ {\matrix{ {Tx} \cr {Ty} \cr {Tz} \cr } } \right] where R is the rotation matrix, representing the trigonometric transformation combination formula of rotation angle around X, Y and Z axes; and T is the horizontal shift vector, which can be transformed into a homogeneous form, as shown in Eq. (2). (2) [XcYcZc1]=[RT01][XwYwZw1] \left[ {\matrix{ {{X_c}} \cr {{Y_c}} \cr {{Z_c}} \cr 1 \cr } } \right] = \left[ {\matrix{ {RT} \cr {01} \cr } } \right]\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr 1 \cr } } \right]

When solving the 3D coordinates of an object, as long as the reference object is selected and the position is fixed, a camera corresponds to a certain external parameter. P′ is also at the same point of image coordinates and pixel coordinates, of which the transformation mathematical expression is shown in Eq. (3). (3) Xp=XIdx+X0Yp=YIdy+Y0 \matrix{ {{X_p} = {{{X_I}} \over {dx}} + {X_0}} \cr {{Y_p} = {{{Y_I}} \over {dy}} + {Y_0}} \cr }

After the image points are collected by the camera, they are projected onto the imaging surface, and the computer stores them in the form of a two-dimensional array, where they are referred to as a pixel point; and each value of the two-dimensional array is called a pixel, which can be converted into matrix form as shown in Eq. (4). (4) [XpYp1]=[1dx0X001dyY0001][XIYI1] \left[ {\matrix{ {{X_p}} \cr {{Y_p}} \cr 1 \cr } } \right] = \left[ {\matrix{ {{1 \over {{d_x}}}0{X_0}} \cr {0{1 \over {{d_y}}}{Y_0}} \cr {001} \cr } } \right]\left[ {\matrix{ {{X_I}} \cr {{Y_I}} \cr 1 \cr } } \right]

At this point, the conversion relationship between the four different coordinate systems can be connected through the camera coordinate system and the image coordinate system, and the conversion between these two coordinate systems is obtained under the ideal pinhole camera imaging model.

The P point of the object is imaged at the P′ point through the lens, and the actual image is on the plane behind the lens. However, for convenience, we move the focal plane to the front of the lens’ optical centre, and the distance between the camera optical centre and the origin of the image coordinate system is the focal length value, f, of the lens. The conversion formula of the two coordinate systems can be obtained by the principle of geometric knowledge, as shown in Eq. (5). (5) XLXc=fZcYIYc=fZc \matrix{ {{{{X_L}} \over {{X_c}}} = {f \over {{Z_c}}}} \cr {{{{Y_I}} \over {{Y_c}}} = {f \over {{Z_c}}}} \cr } which can be expressed in form of matrix as follows: (6) Zc[XIYI1]=[f0000f000010][XcYcZc1] {Z_c}\left[ {\matrix{ {{X_I}} \cr {{Y_I}} \cr 1 \cr } } \right] = \left[ {\matrix{ {f000} \cr {0f00} \cr {0010} \cr } } \right]\left[ {\matrix{ {{X_c}} \cr {{Y_c}} \cr {{Z_c}} \cr 1 \cr } } \right]

Combining the above formulas, the conversion formula of point P in spatial coordinates projected to P′ in pixel coordinates is as follows: (7) ZC[XpYp1]=[1dx0X001dyY0001][f0000f000010][RT01][XwYwZw1]=[fx0X000fy0Y000010][RT01][XwYwZw1]=M[XwYwZw1] {Z_C}\left[ {\matrix{ {{X_p}} \cr {{Y_p}} \cr 1 \cr } } \right] = \left[ {\matrix{ {{1 \over {{d_x}}}0{X_0}} \cr {0{1 \over {{d_y}}}{Y_0}} \cr {001} \cr } } \right]\left[ {\matrix{ {f000} \cr {0f00} \cr {0010} \cr } } \right]\left[ {\matrix{ {RT} \cr {01} \cr } } \right]\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr 1 \cr } } \right] = \left[ {\matrix{ {{f_x}0{X_0}0} \cr {0{f_y}0{Y_0}0} \cr {0010} \cr } } \right]\left[ {\matrix{ {RT} \cr {01} \cr } } \right]\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr 1 \cr } } \right] = M\left[ {\matrix{ {{X_w}} \cr {{Y_w}} \cr {{Z_w}} \cr 1 \cr } } \right] where M is a 3 × 4 matrix, which represents the internal and external parameters of the camera. To determine their specific values, it is necessary to calibrate the camera, and the accuracy of its values will affect the accuracy of numerical solution of 3D coordinates.

2.2.2

Non-linear model

The above situation is a pinhole camera imaging model, that is, a linear model, where the imaging coordinates are linearly related to the coordinates of spatial physical points. However, due to the lens process and the installation deviation between sensors and lenses, the lens system will be distorted, which often leads to the non-linear correspondence between points and a distortion in the imaging system [14, 15]. For this distortion problem, in this paper, a distortion coefficient is introduced to correct it, which is transformed into a linear model.

The lens system usually has two forms of distortion, as shown in Figure 4, which are radial variation along the circle diameter direction and tangential variation along the circle circumference direction. When the ideal imaging point is located at point A because of the distortion, the position of the ideal point is shifted, and the actual imaging point is point B. In Figure 4, dr is radial change and dt is tangential change.

Fig. 4

The radial distortion of the image not only affects the visual effect of the image but also increases the difficulty of calculation in 3D coordinates after feature point matching [16]. In reality, most lenses are circularly symmetric, and the distortion changes with the distance from the centre of the circle. Generally, Taylor series near the distortion centre is used to expand the first three terms [17]. Thus, the formula of distortion of a point in the radial direction is shown in Eqs (8) and (9). (8) XP'=XP+(XP−Xo)(K1R2+K2R4+K3R6) X_P^\prime = {X_P} + \left( {{X_P} - {X_o}} \right)\left( {{K_1}{R^2} + {K_2}{R^4} + {K_3}{R^6}} \right) (9) YP'=YP+(YP−Yo)(K1R2+K2R4+K3R6) Y_P^\prime = {Y_P} + \left( {{Y_P} - {Y_o}} \right)\left( {{K_1}{R^2} + {K_2}{R^4} + {K_3}{R^6}} \right)

Eq. (10) shows the change of coordinate position before and after distortion, where (X_o, Y_o) is the centre point of distortion. K₁, K₂, K₃ are the distortion coefficients. Among them, R represents the Euclidean distance between the point before distortion and the centre of distortion. Tangential distortion is caused by the misalignment between the plane and the sensor plane, which causes the image to have errors in the tangential direction. (10) R=(XP−Xo)2+(YP−Yo)2XI'=XI+[2P1XI+P2(R2+2XI2)] YI'=YI+[2P1YI+P2(R2+2YI2)] \matrix{ {\;\;R = \sqrt {{{\left( {{X_P} - {X_o}} \right)}^2} + {{\left( {{Y_P} - {Y_o}} \right)}^2}} } \hfill \cr {X_I^\prime = {X_I} + \left[ {2{P_1}{X_I} + {P_2}\left( {{R^2} + 2X_I^2} \right)} \right]} \hfill \cr {\;Y_I^\prime = {Y_I} + \left[ {2{P_1}{Y_I} + {P_2}\left( {{R^2} + 2Y_I^2} \right)} \right]} \hfill \cr } where P₁ and P₂ indicate the tangential distortion coefficients. To sum up, there are five distortion coefficients, which can eliminate the distortion and transform the non-linear camera model into a linear model.

Assuming that the position of the point in the object space is determined, the coordinate information of the point in the pixel coordinate system has also been determined. From Eq. (10), it can be found that this is a linear equation set consisting of two equations, but it contains three unknowns, and the equation set has no definite solution from the knowledge of mathematical equations. Only by finding another set of equations in the same form can the accurate information be determined, that is, only when two cameras image the same object can the 3D coordinate information of space object points be determined.

The coordinates of the pixel points of the same object point in two images can be found by technical means, and the 3D coordinates of the space point can be determined by mathematical calculation after obtaining two sets of equations. However, as the calculation procedure associated with this method is complicated, we usually calculate the positional deviation of image points, that is, parallax; then the depth information can be obtained by using the similar triangle principle, and finally the above equations are substituted to obtain the 3D coordinate information [18]. It can be seen from Figure 5 that any point in space will be at the intersection of two straight lines, and that this point is unique. Assuming that the two cameras are of the same model, the optical centre distance is T and the focal length is f; so, the depth information Z can be calculated according to the similar triangle principle, and the calculation formula is shown in Eq. (11). (11) T−(XL−XR)Z−f=TZ {{T - \left( {{X_L} - {X_R}} \right)} \over {Z - f}} = {T \over Z}

The depth information, namely the vertical distance Z, can be obtained from Eq. (12): (12) Z=fTXL−XR=fTd Z = {{fT} \over {{X_L} - {X_R}}} = {{fT} \over d} where X_L − X_R is the deviation value of the same point of an object relative to the coordinate origin in different images, which is regarded as parallax d. Owing to the existence of positive and negative values of the coordinate axis, combined with the different positions of the spatial points of the three objects in the figure, depth information of the spatial points can be ascertained based on Eq. (12), and it can also be inferred that there is an inverse proportional relationship between depth information Z and parallax d.

Fig. 5

The errors that affect the visual matching effect include the measurement of the distance between optical centres, corner detection position error and feature point detection and matching in camera construction. This paper mainly studies the influence of the distance between cameras and the measured depth information on Z-axis coordinate points, the distances between cameras are set as 50–70 mm and the depth information of the same position is measured every 5 mm. Next, the distance between cameras is fixed, and the errors between the measured distance and the real distance are compared at every 100 mm between 100 mm and 5000 mm from the camera. The partial results are shown in Table 1:

It can be concluded from the table that there is an optimal measuring distance between two cameras, and when the camera distance is about 65 mm, the measuring accuracy is the highest. At the same time, it can also be found that the depth and parallax are inversely proportional and non-linear. When the parallax is very small, the small change can causes a large change in the measurement distance, while when the parallax is very large, the change of small parallax will not cause the change of measurement distance.

Under the constraint of inherent conditions, the error of corner detection position mainly lies in the error of pixel position during detection. Since the corner has a certain size, and the number of pixels in the digital image is large and the actual length is small, it may not be the central position when the position is determined, and the actual length corresponds to an offset. Therefore, the error of the calculated internal and external parameters can be effectively analysed.

Corner test is implemented by matching algorithm. First, the 9 × 9 window is used to select the pixel value of the left image, and the pixel value of the right image is calculated by the same calculation method, so as to further compare the sizes of two pixel values. First, when the difference between the two values is less than a certain threshold, it is considered that the initial matching point has been found, the pixel coordinates of the centre point are recorded and all pixel points are matched in turn. Then, we calculate the sum of all pixel values of the matching points of the two images, and take the sum and difference of the pixels of the two images as a cost function. Finally, according to the value of the cost function, we determine whether the next re-matching calculation is needed and if the matching points need to be polished, re-select the starting position of the window according to the form of genetic mutation learning and then determine the centre point to start the next cycle until the matching cost function meets the requirements to finish the calculation and output the best matching result.

The selection of the initial window starts with the origin of the left image, and the matching of the right image also starts with the origin, except that each movement of the left image corresponds to six window movements of the right image. That is, every time the left image moves, the right image moves 2, 6 and 12 pixels in the right and down directions, respectively. Afterwards, the sum of pixel values in the window is calculated, and the threshold value is compared to determine the centre position of the matching start. Similarly, after the right image is moved for six times, the left image is moved for the next round, and the moving mode is to move 2 pixels to the right and down each time.

From Figure 8, we can see that the detection results based on ordinary visual matching have a high degree of confusion and many error points are detected. In contrast, the detection results after considering inherent constraints are very accurate, the accuracy rate is increased from 60.1% to 85.7% and almost all the detected points are corner points. Further statistics are computed based on the test results indicated in Table 2, where the experimental results are analysed and explained in detail using specific data.

Fig. 8

Visual matching under inherent constraints filters out most pixels that are not corners through pre-screening, thus reducing the probability of errors and effectively controlling the number of detected wrong corners. The original model can also detect 54 correct corners, and the performance of the two algorithms is similar in detecting correct corners. However, in terms of the detected wrong corners, by considering the inherent constraints, only three wrong corners were detected, while the original model detected 26 wrong corners, which was about nine times that of the improved algorithm.