Research on face feature point detection algorithm under computer vision

A human feature point is a set of points with special semantics, such as cheeks, eyebrows and lips, on a face image. For the sake of exposition, the set composed of all face images is denoted as II = {I} in this chapter.

Following the customary terminology in the field of face feature point detection, in this paper, we refer to the vector S formed by combining the coordinates of all the face feature points on I as the shape of the face on the image, or, without ambiguity, we simply refer to it as the shape. For ease of exposition, this section first abstracts the formal definition of face shape from the customary symbolic notation:

Definition 1 (Face Shape) For a given p – feature point labeling approach, let x_i ∈ ℝ ² denote the coordinates of the i rd feature point on I, and the vector S = (x₁, x₂, …, x_p) ∈ ℝ ^2p denote the entirety of the p feature points on I.

Face feature point detection is usually learned on a given dataset to obtain the corresponding model, and the average face shape usually has a special meaning in a carefully structured dataset. For the sake of the narrative of this chapter, this section gives definitions of the face feature point dataset and the average face shape:

Definition 2 (Face Feature Point Dataset and Mean Shape) A face feature point dataset consists of a series of images and the face shapes on those images, denoted as D = {(I_i, S_i) | i = 1, 2, …, N}. The mean of the face shapes in D is called the mean shape, denoted as S¯=∑i=1NSi/N .

In a standard face feature point dataset, the sample capacity is usually large, and the angles and poses of the faces in the image are widely and uniformly distributed, so S¯ has a statistically significant average meaning and can usually be used as a standard normal face shape. The standard 68 points in the 300-W dataset available on the i-bug website is the average shape on the 300-W dataset. Later, S¯ is used directly as the standard face shape without further elaboration.

In algorithms of the cascade regression class, face shape-indexed features are often used to describe information about how well the current face feature points match the image. Such features are constructed with the help of face shape indexing of pixel coordinates, i.e., for different face shapes, the positions of these pixels should have similar semantics. For example, when the k pixels used have coordinates p₁, p₂, …, p_k in the coordinate system of the average shape S¯ , and they have coordinates p′1,p′2,…,p′k with respect to any given shape S, the coordinates of p′1,p′2,…,p′k with respect to S should be the same as the coordinates of p₁, p₂, …, p_k with respect to S¯ .

On different face images, after shape indexing, the positions of the corresponding points relative to the face shapes are geometrically invariant and have some semantic information.

Shape indexing can be realized with the help of various geometrical means. For the sake of exposition, in this paper, the process of mapping a set of reference points p₁, p₂, …, p_k to the corresponding points p′1,p′2,…,p′k by shape S indexing is abbreviated as 1p1,p2,…,pk→sp′1,p′2,…,p′k

In the ERT algorithm, the pixel difference feature of shape indexing is proposed by calculating the difference of gray values between shape indexed pixel points under the condition of (1).

The regression-based face feature point detection algorithm can be divided into two parts: feature extraction from the input image and mapping the extracted features to face feature points by a learned regressor. However, due to the complexity of the distribution of feature points, it is very difficult to learn a strong regressor to directly map the features to the exact location, and the results are often unsatisfactory. Therefore, drawing on the idea of integrated learning, the researchers propose to obtain a strong regressor by integrating multiple weak regressors in order to realize face feature point detection. The algorithmic framework can be divided into two parts: the training phase and the testing phase. 1)

Training phase

In the training phase of the algorithm, it can be divided into three main modules: generating training samples, training the optimal weak regressors, and updating the training samples. The framework of the algorithm is shown in Figure 1.

Each training sample can be described as τ=(I,θc,θ^) , where I is a face image, θ_c is the current feature point distribution of the sample, and θ^ is the true feature point distribution of the sample. The generation of training samples only requires randomly generating multiple different initial distributions of feature points θ_c for each set of face images I and corresponding feature point distributions θ^ to obtain the set of training samples T.

After determining the set of training samples T, the pose index features need to be extracted and the weak regressor needs to be trained. For the regressor, its training samples can be denoted as s = (x, δ_θ), where x is the extracted pose-indexed features and δ_θ is the residual from the current feature point distribution to the true feature point distribution. Therefore, the training of the regressor is essentially to learn a method f based on the set of training samples S, for each of which s = (x, δ_θ), satisfies δ_θ = f(x). Similar to the regressor, the weak regressor also needs to learn the regression equation f based on the samples, but the difference is that the weak regressor does not require to absolutely satisfy δ_θ = f(x), but rather requires to satisfy D(f(x)) < D (δ_θ), of which the method D is used to measure the magnitude of the residuals. In other words, the weak regressor can make the current feature point distribution closer to the real feature point distribution as long as it can. The choice of weak regressor is not unique, different regression algorithms can be selected. In the training process, different parameters can be selected to train different weak regressors. The algorithm needs to find the optimal weak regressor during the training process. Remember that the distribution of feature points after the update of weak regressor R on sample τ is θtR , then the optimal weak regressor can be defined by equation (2): 2R*=argminR∑τ‖ θ^t−θtR ‖2,τ∈T

Where θ^ denotes the real face feature point distribution. Eq. (2) represents the selection of the weak regressor that minimizes the error sum of all training samples after updating as the optimal weak regressor weak. The specific details of the extraction of pose index features and the training of the weak regressor will be presented later.

After completing the training of the optimal weak regressor R*, each training sample τ ∈ T, adjusted to τ1=(I,θτR*,θ^) , constitutes a new set of training samples T₁. The optimal weak classifier is retrained using T₁. Repeat this process until all sample errors are below a set threshold.

Figure 1.

Training block diagram of cascade regression algorithm

Figure 2.

Test block diagram of cascade regression algorithm

Pose-indexed feature is a feature extraction method, compared to common feature extraction algorithms such as HOG, SIFT, etc., the most important feature of pose-indexed feature is that it contains the information of pose change in the process of feature extraction. For face images, this pose change information is implicitly included in the face feature point distribution. The pose indexing features are the key to the cascade regression framework to achieve good results, and the overall robustness of the algorithm can be well improved by designing reasonable pose indexing features.

Formally, the pose-indexed feature extraction operator for a face image can be defined as h : I × Θ → Rⁿ. where Θ denotes the set of face feature point distributions, I denotes the input face image, and n denotes the dimension of the output features. From the definition of pose indexed features, it can be seen that numerous different pose indexed features can be extracted for an input face image and a specified set of feature point distributions. Not all pose index features are valid, according to the weak invariance assumption, as long as the designed pose index features satisfy the weak invariance to some extent, the cascade regression algorithm can converge to an accurate result after a certain number of iterations.

Weak invariance is a property of pose-indexed features, which indicates that consistent pose-indexed features can be extracted from images with different poses. In order to facilitate the description of weak invariance, this paper defines the camera mapping method as C : R³ × T → R², which indicates that a 3D coordinate point p_r ∈ R³under a head coordinate system under a rotational change t ∈ T is obtained by camera mapping to a pixel point p_i ∈ R² on a 2D plane, and then the face image captured under pose t is noted as I_t. Then a set of face feature points P_f ⊂ R³ under a specific pose t can be represented as θ = C(P_f, t) after mapping to a 2D image, so that a shape index feature with weak invariance The shape-indexed features can then be represented as satisfying Eq. (4) for any t₀, t₁ ∈ T. 4h(θt0,It0)=h(θti,Iti)

That is, satisfying weak invariance means that consistent features can be extracted from face images with different poses for the same set of feature points P_f under 3D space.

Considering that pose-indexed features will be computed very frequently during the training and testing process of the whole cascade regression algorithm framework, the design of features should also be as simple as possible to ensure that the algorithm can realize real-time face feature point detection. Based on the pixel difference features, the algorithm first randomly selects P points as reference points on the face image, and then estimates the rotation and scaling parameters according to the current distribution of face feature points to transform the reference points. Finally, the pixel difference between the two transformed reference points is utilized to form a lissomorphic feature of dimension P². The key to the weak invariance of the pixel difference feature is to transform the reference points using the rotation and scaling parameters. However, since the pixel difference feature is obtained directly using the global coordinates, the reference point will be larger in the case of a large expression transformation or head pose transformation, in order to improve this problem, the global coordinates of the reference point are no longer used directly, but the relative coordinates between the reference point and its closest neighboring feature point are used to determine the reference point. Based on this, a further improvement is made by randomly selecting the reference point on the line between two feature points. The experimental results show that all these improvements somewhat enhance the robustness of the image soliton difference feature to head motion.

The data distribution statistics are used to train all levels of layer regressors, so that the cascade regression framework can complete the incremental learning, which can gradually correct the initial shape to the position of the real shape marking, and improve the accuracy of face feature point localization. While the incremental learning needs to be built on an existing offline model to be carried out, this section first introduces the learning process of the offline model.

The N image and the corresponding real shape markers in the training set are denoted as I = {I}_i=1⋯N and S*={ si* }i=1…N respectively, where si*=[ x1,y1,…,xl,yl,…,xL,yL ]T is used to denote the position vector of the L feature points and the coordinates of the l th feature point are denoted as [x_i, y_l]. All the samples are initialized and denoted as S0={ si0 }i=1…N , which is usually set to be the average of the vectors of all the real shape markers in the training set. In this section of the method, SIFT features are used to extract local features sequentially based on the order of all feature points and spliced to form shape index features, denoted as Φ(I, s_i) ∈ ℝ ^d. The goal of the cascade regression is to model the mapping relationship between the shape index features Φ(I, s_i) and the shape displacements δsit=sit−1−si* , such that the initial shapes are progressively corrected to the positions of the true shape markers. The following equation shows the optimization expression in cascade level t: 5argminGt∑i=1N‖ δsit−Gt(Φ(I,sit-1)) ‖22 where G(·) denotes the extreme learning machine that regresses the shape. Let x^t = Φ(I, s^t–1), then the regression function can be expressed as Gt(x)=∑k=1KβkG(ak,bk,xt) , where K is the number of hidden layer nodes of the ELM. The current shape displacements of all samples are represented using a matrix as ΔSt=[ δs1t,…,δsNt ]T , and the optimization problem in the above equation is rewritten as: 6β˜t=argminβt||ΔSt−Htβt||22

Where Ht=H(a1t,…,aKt,b1t,…,bKt,x1,…,xN) is calculated and the Sigmoid function is chosen as the activation function of the neuron with the expression: G(a,b,x)=11+exp(−(a⋅x+b)) . Equation (6) is solved to obtain the parameters of the ELM network for the t rd level layer Gt=[ at,bt,β¯t,M′ ] and saved as a regressor. The amount of shape displacement is calculated by the trained regressor and the current shape is updated using equation (7): 7sit=sit−1−Gt(Φ(Ii,sit−1))

The learning process of traditional cascaded homoscedastic methods requires iteratively training each level of the hierarchical regressors until the error between the current shape and the true shape is no longer increasing. However, such a sequential approach makes learning the model longer and does not allow for incremental updates of each regressor. As shown in FIG. 3, since the shape displacement space δs^t in the level t layer is completely dependent on the computed results of the level t – 1 layer regressors. If a new sample {δs_new, x_new} is entered for incremental learning, the regressors in the first level can be easily updated using the ELM’s incremental learning approach to generate a new regressor Gnew1 . All samples, including the old and new samples { { si1 }i=1N,snwe1 } , need to be recalculated using the updated regressors Gnew1 . Therefore, the regressor for the second level must be recalculated for all samples in the output of Gnew1 . Similarly, the regressors for all subsequent levels are updated in a similar manner. Therefore, the incremental update procedure for the back-to-sun hybridizer sequential training approach is very time-consuming and computationally resource-intensive. To solve this problem, the approach in this section utilizes sampling to generate a shape displacement space from a known mixed Gaussian distribution, which replaces the shape displacement space obtained from the previous cascade level computation. As shown in Fig. 4, this strategy can effectively realize that all regressors in the cascade framework can be incrementally updated in parallel when needed to improve the learning efficiency. The statistical parameters of the mixed Gaussian distribution of shape displacements in each cascade layer can be collected through the sequential training approach described above, in the form of the mean and variance of the shape displacement space.

Figure 3.

The training manner of conventional CR

Figure 4.

Parallel update method based on mixed Gaussian distribution sampling

Specifically, the input space of the level t layer contains N shape displacement vectors { δsi }j=1N and since the shape of faces captured in natural environments is usually oriented in the unquestioned direction, it is assumed that there exist J Gaussian distributions describing the input space. The shape displacement probability distribution of the sample is assumed to be: 8p(δsit|Θt)=∑j=1JπjNt(δsit|μj,Σj) where π_j, j = 1, ⋯, J denote the probability that the shape displacement vector δsit belongs to the j th subdistribution, and N(δs_i|μ_j, Σ_j) denotes the j th Gaussian distribution function and the mean and variance of its distribution are μ_j and Σ_j, respectively. Therefore, the set of Gaussian mixture distribution parameters in the t th level layer is denoted as Θ^t = {π₁, π₂ …, π_J, μ₁, μ₂, …, μ_J, Σ₁, Σ₂, …, Σ_J}, which is the mean, variance, and probability of each subdistribution occurring in the mixture model. Learning the model parameters in Eq. (8) is typically solved using an iteration-based EM algorithm designed to iteratively solve for the expectation and model parameter maxima until logp(ΔSt|Θt)=log∏i=1Np(δsit|Θt) converging to a local minimum, where ΔSt={ δs1t,δs2t……δsNt } , the solution process is briefly described below (superscripts denoting the number of layers will be omitted t for brevity of writing).

Step E – : Fix the parameters and solve for the a posteriori probability, i.e., the normalized probability that the shape displacement vector δs_i on the i th comes from the jth distribution, which in step m is solved by Eq. (9). 9zji=πjmp(δsi|μjm,Σjm)∑IJπlmp(δsi|μlm,Σlm)

Step M – : The model parameters Θm+1={ π1m+1,m2m+1,⋯,πJm+1,μ1m+1,μ2m+1,⋯,μJm+1,Σ1m+1,Σ2m+1,⋯,ΣJm+1 } are estimated through the estimates of zji and the constraints ∑j=1Jπj=1 , where μ ∈ R^2L, Σ ∈ R^2L×2L. These distributional parametric quantities are updated through equation (10): 10πjm+1=1N∑i=1Nzji,μjm+1=∑i=1N=ijsi∑j=1N=ijiΣjm+1=∑i=1Nzji(δsi−μjm+1)(isi−μjm+1)T∑i=1Nzji,

The EM algorithm solves for the hybrid Gaussian distribution to better characterize the distribution of shape displacement vectors in the level t layer. On the HELEN dataset, the trained shape displacement vectors and the computationally generated hybrid Gaussian distribution (K = 4) can more accurately quantify the current distribution of shape displacement vectors than the single Gaussian distribution generated in the face shape space. The distribution statistics {Θ¹, Θ², …, Θ^T} of the T hierarchical layers are collected during the offline training process.

With the known distribution statistics of the data in the T hierarchical layers, all the hierarchical regressors can be updated in parallel based on these distribution statistics. As shown in Fig. 4, the input space of shape displacement is generated by sampling all the cascade layers with known data distributions, and all the extreme learning machines run in parallel, which not only improves the efficiency of learning but also facilitates the incremental learning of the whole cascade regression framework.