Research on Gaze Estimation Method Combined with Head Motion Changes

The current gaze estimation methods are mainly divided into two categories: feature-based gaze estimation and epigenetic gaze estimation. Feature-based sight estimation methods can be divided into model-based and regression-based sight estimation methods according to different implementation schemes of eye direction and eye feature mapping relationship [1].

The model-based gaze estimation method first takes the straight line obtained by connecting the pupil or iris center as the optical axis. At the same time, a fixed deviation Angle between the optical axis and the visual axis is obtained based on prior knowledge to compensate for the actual line of sight to improve the accuracy of the gaze estimation. Chen proposed a 3D gaze estimation algorithm based on single-camera face tracking [2]. The algorithm was implemented without the help of an external light source. The solving equations of the 3D eye center, pupil center, and visual axis were derived, and the fixation point was solved by one-time calibration. Regression-based gaze estimation methods usually includes the following two processes: selecting and extracting eye motion and constructing regression function to obtain the mapping relationship between the line of sight and eye movement features. Eye movement is usually obtained by connecting a fixed point unrelated to eye movement with a moving point strongly related to eye movement. The most commonly used eye movement is calculated by the corner point and the iris center point. Yamazoe proposed a real-time gaze estimation method using specific face feature association. This method calculates the geometric relationship between the eye center and the eye radius in advance and simplifies the sight estimation method based on a 3D sight model by tracking specific facial features [3]. Valenti used the method of Isocenters for iris center location and canthus location under a single webcam and achieved a better detection accuracy in low-resolution images [4].

Appearance-based gaze estimation method extracts features from the whole image and then obtains the mapping relationship model between features and line of sight direction through training. In the early stage, scholars used manual features such as HOG or LBP to extract features and then used K-nearest neighbor [5], random regression forest [6] and support vector machine (SVM) [7], and other models to achieve line-of-sight estimation. Subsequently, with the development of neural network technology, neural network models based on deep learning have been widely applied to gaze estimation. To improve the generalization performance of appearance-based models, Wang proposed a method combining adversarial learning with Bayesian inference for the overfitting of appearance, head pose, and point estimation [8]. To solve the interference of face images with free head movement on the line of sight model mapping, Zhou proposed to input different areas of the face into the 3D line of sight estimator with adaptive weighting during head movement, which greatly improved the efficiency and accuracy of the regression model [9].

Head pose estimation is intrinsically related to visual gaze estimation, namely, the head pose can represent the direction and focus of the human eyes. Physiological studies have shown that a person's gaze prediction comes from a combination of head posture and eye orientation. There are mainly the following methods for obtaining head pose: the method based on standard template matching, the method based on the geometric relationship of key points of the face, the method based on feature regression, and the method based on manifold learning [10].

The method based on standard template matching compares the header image with the model with pose labels in the feature space [11] and calculates the similarity between the input features and the standard template set [12].

The method based on the geometric relationship of the key points of the face determines the corresponding pose according to the position of the eyes, mouth, and nose tip, which can recover the global pose change of the head from the video. Fridman used the random forest machine learning method to classify the pose of the driver's facial 56-point position relationship features [13]. The geometric method is simple, fast, and has low time complexity. However, this method has high requirements for the detection and location of facial feature points.

The multi-layer perceptual neural network is a common regression tool in the method based on feature regression. This method corresponds the output layer of the network to the discrete head posture, and the head posture estimation results are obtained by training the neural network directly. Murphy train regressors by support vector regression (SVR) [14]. Huang introduced the random forest algorithm into head pose estimation [15]. The method based on regression has good real-time performance and high accuracy, but it requires a large amount of computation and is greatly affected by the head detection and positioning results.

Based on manifold learning, Huang improved the accuracy of head pose in non-uniform samples by supervising local subspace learning; and discussed the phenomenon of over fitting [16]. Foytik roughly estimated the head pose through the linear supervision function, and then realized the accurate judgment of the head pose through the linear regression function [17]. The method based on manifold embedding has high time complexity and low accuracy, which is still far from the real practical process.

Before starting the experiment, the camera and other equipment should be calibrated to ensure that they are in normal operation. Subjects e adjust their seated position on the chair to be approximately 60cm away from the center screen, ensuring that they are within the best focal length of the camera and that the origin of the subject's head is as close to the origin of the camera coordinate system as possible.

At the beginning of the experiment, a red dot with a radius of 25 pixels was displayed on the screen. The red dot represented the object to be captured by the eye. Each time the red circle appeared, the subject was asked to look at the center of the red circle and press the space bar at the same time. At this point, the camera would take an image of the subject's face, and the software would record the coordinates of the target point, which would then show the next origin. This is shown in Figure 1. Through the experiment, a set of data, including a face image and screen coordinates, are obtained.

Figure 1.

When the head is still, there is only eye movement in the process of gazing at the target, and the images captured are all frontal face images, which are almost the same, except for the difference in eye direction, as shown in Figure 2.

Figure 2.

Experiments in the dynamic line-of-sight estimation stage are called dynamic experiments, and the experimental environment is the same as that of static experiments. Before starting, subjects should adjust their sitting position so that the distance between subjects and the center screen is about 60cm. At the beginning of the experiment, the subjects' eyes were required to face the center of the middle screen, and the initial state of head posture was calibrated by pressing the space bar. Dynamic experiments require subjects to move their heads in a specific way by focusing their eyes on a static point in the center of a computer screen, and moving their heads while still focusing on that point.

In this experiment, the subject can rotate the head freely, rotate around the X, Y, and Z axes, and generate the pitch Angle, yaw Angle, and roll Angle as well as the translation along the X, Y, and Z axes. A 5-second video clip was collected for this experiment. During head movement, the coordinated movement of the eye and the head occurs during the fixation of the target, and the pictures obtained by shooting show various appearances, resulting in the attitude change of the eye region, as shown in Figure 3.

Figure 3.

To obtain the eye image, the location of the iris center of the human eye and the inner corner of the human eye are extracted to construct eye motivities. We need to extract the position region of the face and the position coordinates of the key points of the face from the image.

The face detection method based on Adaboost and wavelet features are used to obtain the location region information of the face. Based on Haar-like features, the algorithm added a cascade classifier and integral graph algorithm to improve the speed of image processing, meet the real-time operation requirements of the whole fixation point estimation algorithm, and also to ensure the accuracy of face detection. The theory of integral graph is introduced into the algorithm, which can reduce the problem of excessive computation due to the excessive number of Haar features in the search window. The result of face recognition is shown in Figure 5.

Figure 5.

Next, face feature point detection, face feature point detection is also known as face alignment, through the algorithm to detect the inner and outer corner of the face, nose tip, corner of the mouth, and another key points of the face position coordinates. Under normal circumstances, we get the position coordinates of 68 feature points of the face. After using the adaboost+haar-like algorithm to obtain the region location of the face, this paper uses the dlib c++ library to detect the feature points of the face. The position coordinates of 68 feature points of the face were obtained. The main feature points of the face included the feature points around the eye area, the coordinate points of the inner corner of the eye needed to constitute the eye movement, and the important feature points of the face needed to calculate the rotation matrix and displacement matrix of the head. The detection results of face feature points are shown in Figure 6.

Figure 6.

Based on face feature points, this paper uses gradient analysis combined with star rays to calculate the position coordinates of iris center points. The algorithm can be run in the case of low-quality images, which not only ensures the computational speed but also improves the robustness of the algorithm, and can meet the different head movements and the changes of light and shade.

After obtaining the eye movement information, the mapping relationship between the feature information data of eye movement and the position coordinates of points on the screen should be established. The mapping relationship between the feature data of eye movement data and the coordinate points of the screen is a kind of nonlinear mapping relationship. Due to the real-time requirement of the gaze point estimation system, it is necessary to select a mapping model that can process the high-dimensional data and has a fast training speed. After a comprehensive comparison of various mapping models, random forest is chosen as the mapping model.

Through the motion analysis of the head sequence image, the global motion of the whole head is considered as a rigid body motion model, and the head motion parameters are estimated. An object whose 3-D distance between any pair of points on the object does not change with time is defined as a rigid body. The human head can be simplified into a rigid body in the physical sense, that is, the distance between the surface points of the head's rigid body remains unchanged during its movement. Therefore, motion parameters can be estimated based on the property that the distance between points remains unchanged before and after rigid body motion.

Compared with the classical head pose estimation methods, EPNP and POSIT algorithm, it is found that the head pose estimation algorithm based on PNP has smaller advantages in estimation accuracy than the POSIT algorithm, while the head poses estimation algorithm based on POSIT has obvious advantages in calculation speed. Therefore, In this paper, the POSIT algorithm is chosen to solve the head posture, which not only ensures the requirements of solving accuracy but also meets the requirements of computing speed.

The position coordinates of the obtained facial key points and the corresponding key points of the general 3D face model were mapped and projected, and the rotation and displacement matrix (RT) of the final head pose estimation and the Yaw, Pitch, and Roll angles of the final head were calculated.

68 face feature points were obtained, from which several key points with different depths of high reliability and robustness were selected to ensure the accuracy and robustness of head pose estimation. Therefore, 8 key points were selected. That is, the left corner of the left eye, the right corner of the left eye, the right corner of the right eye, the left corner of the right eye, the nose tip, the left corner of the mouth point, the right corner of the mouth point and the lower frontal corner are used to estimate the head pose. The estimation result of the head pose is shown in Figure 7.

Figure 7.

Is by the human eye eye eye gaze position information and joint decision movement of the head, in the head and the screen under the condition of relatively static, estimates the gaze of the initial state, to achieve the head moving cases of the fixation point estimate, need to deal with the movement of the head, above calculated the rotation matrix and translation matrix of the head, The initial fixation position was compensated according to the deflection of the tester's head, and the actual fixation position was finally obtained.

We assume that the influence of head movement on the position coordinates of fixation point is (Δu_x, Δu_y). In the initial case, the head remains stationary between the screen, and the 3D initial coordinate of a point of the head is set as (x₀, y₀, z₀), the pixel coordinates of the projection of this point on the two-dimensional plane is (u₀, v₀), when the head movement occurs, the coordinates of this point are (x₁, y₁, z₁), We obtained the rotation matrix and translation matrix in the case of head movement through the improved POSIT algorithm, and the formula can be obtained as follows: (1) [x1y1z1]=R[x0y0z0]+T \left[ {\matrix{ {{{\rm{x}}_1}} \hfill \cr {{{\rm{y}}_1}} \hfill \cr {{{\rm{z}}_1}} \hfill \cr } } \right] = {\rm{R}}\left[ {\matrix{ {{{\rm{x}}_0}} \hfill \cr {{{\rm{y}}_0}} \hfill \cr {{{\rm{z}}_0}} \hfill \cr } } \right] + {\rm{T}}

Where the R rotation matrix and T translation matrix are shown below. (2) R=[r11r12r13r21r22r23r31r32r33] {\rm{R}} = \left[ {\matrix{ {{{\rm{r}}_{11}}} & {{{\rm{r}}_{12}}} & {{{\rm{r}}_{13}}} \cr {{{\rm{r}}_{21}}} & {{{\rm{r}}_{22}}} & {{{\rm{r}}_{23}}} \cr {{{\rm{r}}_{31}}} & {{{\rm{r}}_{32}}} & {{{\rm{r}}_{33}}} \cr } } \right] (3) T=[ΔXΔYΔZ] {\rm{T}} = \left[ {\matrix{ {\Delta {\rm{X}}} \cr {\Delta {\rm{Y}}} \cr {\Delta {\rm{Z}}} \cr } } \right]

Therefore, the offset caused by the head movement can be calculated by the formula (Δu_x, Δu_y), and is the focal length of the camera. (4) Δux=fx1z1−u0 \Delta {{\rm{u}}_{\rm{x}}} = {\rm{f}}{{{{\rm{x}}_1}} \over {{{\rm{z}}_1}}} - {{\rm{u}}_0} (5) Δuy=fy1z1−v0 \Delta {{\rm{u}}_{\rm{y}}} = {\rm{f}}{{{{\rm{y}}_1}} \over {{{\rm{z}}_1}}} - {{\rm{v}}_0}

The position coordinate of the gaze point under the stationary state of the head is (u_x, u_y), and the position coordinate of the gaze changed by the head movement is (s_x, s_y): (6) sx=ux+Δux {{\rm{s}}_{\rm{x}}} = {{\rm{u}}_{\rm{x}}} + \Delta {{\rm{u}}_{\rm{x}}} (7) sy=uy+Δuy {{\rm{s}}_{\rm{y}}} = {{\rm{u}}_{\rm{y}}} + \Delta {{\rm{u}}_{\rm{y}}}