9D Rotation Representation-SVD Fusion with Deep Learning for Unconstrained Head Pose Estimation

Many existing neural networks use depthwise separable convolution to extract features, and although its structure possesses fewer parameters as well as smaller FLOPs compared to normal convolution, it is usually not able to fully utilize the gas pedal with the available hardware. This paper utilizes EfficientNet V2 as the feature extraction network, which is a more advanced and lightweight convolutional neural network model compared to EfficientNet. It is characterized by low number of parameters, high accuracy, and excellent training and inference speed. A notable improvement is the substitution of EfficientNet’s shallow MBConv with the Fused-MBConv module. The Fused-MBConv module substitutes the expansion 1x1 convolution and depthwise 3x3 convolution in the primary branch of the original MBConv structure with a 3x3 convolution. This solves the problem of employing depth-separable convolution in the initial layer of the network. The issue of slowdown caused by using of depth-separable convolutions in the shallow layers of the network is effectively solved, resulting in an important enhancement in training speed. Figures 2 and 3 show the structure of the MBConv and Fused-MBConv modules, respectively. Table 1 shows the EfficientNet V2-S structure. The method proposed in this paper is adapted by changing the final fully connected layer output to 9.

Figure 2.

MBConv

Figure 3.

Fused-MBConv

Choosing a suitable approach for representing rotation is vital for accurately estimating head posture. Traditionally, Euler angles have been employed. Nevertheless, this method of representing rotation is not ideal because to its susceptibility to gimbal lock. In such cases, specific sequences and angles of rotation can cause the loss of one of the three independent rotation axes. Another rotation representation is the quaternion method, which is not affected by gimbal lock but has the issue of double coverage. This means that for each rotation, there are two corresponding quaternions. While these two representations are physically equivalent, they exhibit a significant numerical discontinuity. Therefore, neural networks struggle to learn accurate poses in the presence of numerical discontinuities.

Figure 4 shows two examples of pictures with comparable visual presentation from the 300W-LP dataset. A comparison of the two pictures shows that the Euler angles and quaternions have distinct labeling values, notably the second value, yaw, in the Euler angles. Positive and negative numbers suggest entirely opposing attitudes. A better rotation representation is the rotation matrix, which is a continuous representation, only the rotation matrix can reflect the similarity of pose appearance. In SO(3) space, the rotation matrix is a 3×3 matrix that satisfies the orthogonality criteria RR^T = I, where R^T represents the transpose of R, and I represents the identity matrix. The R9+SVD method can work with any 3×3 matrix and turn it into a valid rotation matrix in ^SO(3). R9 refers to using the neural network to directly output 9 vector values that describe a 3×3 rotation matrix. The SVD method was created because directly estimating 9 variables might not work because rotation matrices have certain qualities, such as being orthogonal and having a positive determinant.

Figure 4.

Image samples from 300W-LP dataset with different rotation representations

Given a 3×3 matrix, its singular value decomposition (SVD) is expressed as: 1M=UΣVT

Here, U and V are 3×3 orthogonal matrices, and ^Σ is a diagonal matrix containing the singular values of matrix M. To project M onto the rotation matrix ^R,R must satisfy two conditions: 1)

The column vectors of R must be of unit length and orthogonal to each other.

Therefore, after adjusting the singular values, R is constructed as: 2Σ+=diag(1,1,det(UVT))

Reconstruct the rotation matrix using the adjusted singular value matrix: 3R=UΣ+VT

Hereby, ^det(UVT) ensures that the determinant of ^R is 1, while the combination of ^U and V^T ensures that the column vectors of ^R are orthogonal. Therefore, the neural network predicts 9 parameters, which are then transformed into a 3 × 3 rotation matrix while sticking to the orthogonality requirement.

The advantages of this method are: 1)

Smoothness: It provides a continuous and smooth representation, allowing optimization algorithms like gradient descent to converge effectively while avoiding issues such as the singularities of Euler angles or the double coverage problem of quaternions.

The loss function commonly used in previous head pose estimation tasks is the L2 loss function, and the calculation method is equation (4). However, in the head pose estimation task, there are some problems when using the L2 loss function to measure the difference between rotations. First, the L2 loss does not take into account the periodicity of the rotation angle, which makes it impossible to correctly evaluate the similarity between rotations close to 360 degrees or -360 degrees. Secondly, the L2 loss assumes that all dimensional changes are independent and linear, which is inconsistent with the geometric structure of the rotation matrix or quaternion.4loss(x,y)=1n∑i=1n(yi−f(xi))2

The geodesic loss function measures the distance between two rotation matrices along the shortest path on the manifold, known as the geodesic. The geodesic loss function is calculated based on the trace of the rotation matrices, with the formula given as: 5d(R1,R2)=cos−1(tr(R1R2T)−12)

R₁ and R₂ ∈ SO(3), representing the predicted rotation matrix and the true rotation matrix, respectively. The trace (tr) denotes the sum of the diagonal elements of a matrix. This distance will be used as the loss function for the neural network in subsequent experiments.

This work trained and evaluated it method on various types of datasets. The most commonly used publicly available datasets for head pose estimation are 300W-LP [15], AFLW2000[16], and BIWI [17].

The head pose estimate error is measured using the Mean Absolute Error (MAE) of Euler angles, which is the most widely used metric. This is represented by Equation (6).

^N refers to the total number of face images, x_g represents the true values of the head poses, and ^x_p represents the predicted values of the head poses.

This work employed PyTorch to create the whole model, with EfficientNetV2-S serving as the backbone network, and trained the network for 30 epochs with the Adam optimizer. The initial learning rates for the backbone network and the final fully connected layer were set to 1e-5 and 1e-4, respectively, with each learning rate halving every 10 epochs. The batch size was set at 64.

In the first experiment, the network was trained using the synthetic 300W-LP dataset and subsequently tested on two real-world datasets: AFLW2000 and BIWI. The evaluation metric used was the mean absolute error (MAE) of Euler angles, which required transforming the predicted rotation matrices into Euler angles for comparison purposes. Table 2 presents the results of the first experiment, comparing the proposed approach to other state-of-the-art landmark-free head pose estimation methods. The experimental results demonstrate that the proposed strategy outperformed the current best methods by approximately 22% and achieved the lowest error rates in pitch, yaw, and roll angles on the AFLW2000 dataset. On the BIWI dataset, the approach exceeded seven out of eight of the most advanced algorithms in terms of MAE. Figure 5 illustrates the results of the method on the AFLW2000 dataset after converting the predicted rotation matrices to Euler angles.

Figure 5.

Example images of Euler angle visualization using rotation matrix transformation from AFLW2000 dataset

In the second experiment, the method outlined by FSA-Net was followed, with the BIWI dataset randomly split into training and testing sets in a 7:3 ratio. The results were compared with other networks that employed the same experimental approach. Table 3 presents the results of the second experiment. The proposed method outperforms other methods in terms of MAE, and shows superior performance in yaw and pitch, with roll being better than most. These experimental results demonstrate the robustness of the proposed method, as it achieves stable and accurate results in both Euler angle and MAE across different datasets.

To demonstrate the superiority of the geodesic loss function as a distance metric for head pose estimation, additional tests were conducted using the rotation matrix. To support this claim, the previous experiments were replicated by training the network with the L2 loss function. Table 4 presents the effectiveness of the proposed technique when trained with two distinct loss functions. Training the network with the geodesic loss function yields superior results compared to the L2 loss.

This paper also examines the influence of different backbone networks on the results obtained by employing geodesic loss. In order to do a comparison, this research employed the ResNet [21] network as an illustrative example. The findings shown in Table 5 demonstrate that the approach outlined in this research achieves exceptional results when used to the EfficientNet V2-S backbone network. By employing ResNet18 as the foundation network, this approach surpasses the majority of previous approaches in terms of performance on both the AFLW2000 and BIWI datasets. This illustrates that employing a suitable rotation representation greatly enhances the accuracy of head pose estimation.

In the final experiment, the THOP library was used to compare the proposed method with 6DRepNet in terms of parameter count and floating-point operations (FLOPs). As shown in Table 6, the proposed approach achieved a lower MAE while requiring fewer parameters and FLOPs.