Construction of an Accurate Tracking and Ai Evaluation System for Dance Movements by Incorporating Image Recognition Technology

The human body posture-based analysis system’s acquisition steps are depicted in Fig. 1, which mainly includes industrial camera, switch, computing workstation, and human-computer interaction interface. The industrial camera is used for image acquisition, the camera is connected to the switch through a network cable, and the computing workstation is connected to the switch through a network cable, thus realizing the transmission of image data from the camera to the computing workstation.

Figure 1.

Human attitude analysis system

In recent years, artificial intelligence technology has been advancing by leaps and bounds, and AI technology has been applied in the fields of color selection, medical testing, unmanned driving, security, and so on. These technologies cannot be separated from the image acquisition. These technologies are inseparable from the acquisition and processing of images. The recognition process of the industrial camera system involves taking a picture of the target object and passing it to the switch through the network port. After the picture algorithm processing algorithm to obtain the target area. Dance action is in constant motion, and industrial cameras can continuously capture multiple frames of dance action at different moments.In this way, the image of the fruit can be acquired in all directions and without a dead angle.The captured images are transferred to the memory buffer of the industrial computer and processed by a specific algorithm to obtain the external characteristics of each fruit.Prepare for the later hit sorting.

To capture high-quality images of dance action at high resolution, it is necessary to study high frame rate, high resolution, low-cost industrial cameras, which are mainly divided into CMOS or CCD cameras.With the development of semiconductor technology, compared to the CCD industrial camera, CMOS image industrial cameras have greatly improved in terms of resolution, power, cost, and vibration resistance.So this paper studies the use of CMOS industrial cameras to capture images of dance movements.The camera’s image acquisition function allows for the collection of three-dimensional data on 25 skeletal joint points in the human dance posture. Based on the 3D data, the human skeletal structure is established, the depth information of the human body is replicated, the dynamic change of the 3D data of the skeletal joint points under the constant change of the dance posture is tracked in 3D space, and the 3D data set of the human dance posture movement is generated for subsequent research.

The full name of CTC is connectionist temporal classification, which is a classification model for temporal sequences. Considering the video as a continuous image sequence that preserves the human body pose information, the action sequence is segmented based on the trend of the human body’s major joint positions in consecutive frames. In the part of human action segmentation based on captured image data, with the idea of CTC, an end-to-end approach is used to generate the prediction results directly from the original data, thus transforming the segmentation problem into the annotation problem of the sequence.CTC can automatically determine the segmentation points, eliminating the need for human annotation of the samples, which greatly improves the efficiency of the segmentation, and at the same time, avoids the cumbersome feature extraction process in the traditional methods. The human action segmentation model based on CTC proposed in this section consists of three parts: GRU neural unit, logistic regression, and CTC Loss, of which the GRU neural unit is an important basic construction, which can avoid the long-term dependence problem of ordinary RNN units and requires fewer parameters and less training time than LSTM. And the true label sequence of the action sequence to calculate the CTC Loss. During training, the model updates the parameters by differentiating the CTC loss and updating them accordingly.The input to the model is in terms of human action time sequences, and the output is the probability that each frame in that action sequence belongs to each label or category.

For the sequence labeling task, assuming that all of its labels constitute the alphabet A, the number of softmax output layer units of CTC is one unit more than the alphabet A labels. The first |A| unit denotes the probability of belonging to its label while the last unit denotes the probability of belonging to space (blank) and not belonging to any of the labels. Define some notations before formalizing the principle of CTC. Define the extended alphabet A′ = A ∪ {blank} with the network activation value ykt representing the probability that the output sequence has an output label of k at step t. Assuming that each time step is independent of each other then, the probability that the output path is π in case the input sequence is x: (1) p(π|x)=∏t=1T(ykt)

With F representing a many-to-one mapping, a transformation that maps an output path π to a sequence of labels l. For example, F (a–ab–) = F (–aa––abb) = aab, where - represents the space (blank). Thus the probability that the output sequence is l can be expressed as the sum of the probabilities that the sequence of all output paths π after mapping F is l, expressed as Eq: (2) p(l|x)=∑π∈F−1(l)p(π|x)

Mapping F can convert numerous different output paths into the same labeled sequence, a step made possible by CTC’s use of unsegmented data, which allows the network to predict labeled sequences with unknown segmentation points.

The analysis of human movement posture based on the acquired images is to estimate the movement posture of the human body from one or more viewpoints. In order to analyze the dance movement posture data, this paper uses feature points to calibrate the joint points of the human body, and the human skeleton is regarded as a multi-rigid-body connected model, and the connecting line between the feature points represents a rigid body, so that there is no deformation between the adjacent joint points at any moment. The main action postures are the movements of the head, torso, hips, and limbs.

The movement of human body is a complex process, without considering the role of muscles, nervous system and other conditions, the human body movement can be abstracted as a simple chain system movement connected by some rigid bodies. Taking Coor Joint to represent the absolute position coordinates of the feature point in the world coordinate system, the upper limb is composed of two rigid bodies of upper and lower arms connected by the elbow joint, and the lower limb is composed of two rigid bodies of thighs and calves connected by the hip joint, and the thighs and calves are connected by the knee joint. The head, torso, and hips are represented as one rigid body by a line connecting the joint points. With the rigid body structure of the human body described earlier, the spatial coordinates of the marker points are used as the data basis for calculating the vectors, which serves as the basis for the similarity matching analysis that follows.

After fully considering the complexity and professionalism of its dance movements, in order to more accurately assess the standard degree of dance movements, this paper divides the 24 joint points of the human body captured by the posture estimation into regions, and the division regions are shown in Fig. 2, which represent the LH of the left hand, the RH of the right hand, the LL of the left leg, the RL of the right leg, the 5 core torso regions of the UB of the upper body, and the 1 lumbar-abdominal center Root region, respectively. Practicing dance requires an understanding of the movement of each joint and its relationship with the parent joint. Therefore, the core torso regions need to be divided from global to local positions. In this paper, we propose the GL-Compound method, which introduces the concepts of global motion parameter (GMP) and local motion parameter (LMP), and divides the limbs and the core regions of the upper body with respect to the center region of the body into the global motion of the limbs and the limbs at the end of the body and the local motion of the skeletal spatial position are superimposed to represent the dance movements.

Figure 2.

GL-Compound physical activity area division

The motion parameters of the left and right arm parts are calculated in the same way, and in the case of the right hand RH, for example, its motion can be regarded as a superposition of the motion of two parts, one part of which is the global motion of the plane S₂, which consists of the greater and lesser arms, with respect to the center plane of the body S₀, and the other part of which is the local motion of the palm of the hand S₆. In this example, with the root joint kıs as the origin in space, a Cartesian coordinate system is established to compute the global Euclidean distances between the shoulder joint k₁, the elbow joint k₂, and the wrist joint k₃ in the global plane and the root joint in 3D space, respectively, and at the same time solving for the angle between the normal vector of this global plane S₂ and the normal vector of the center plane S₀ to obtain is the GMF parameter of the right arm RH. For the ten local parameters LMP, it is necessary to ask for the local Euclidean distances between the middle finger k₄, the thumb k_s and their parent wrist joint k₃, while solving for the angle between the normal vector of the local plane S₆ and the upper global plane S₂: (11) d(p1,p2)=(x2−x1)2+(y2−y1)2+(z2−z1)2

Equation (11) solves for the Euclidean distance between joints in three-dimensional space, with P₁ and P₂ being the three-dimensional coordinates of the two joints, respectively.

Any plane is formed by three joints, assuming that the coordinates of the joints are P₁(x₁,y₁,z₁),P₂(x₂,y₂,z₂),P₃(x₃,y₃,z₃), then the plane normal vector can be obtained by calculating the product of two vectors induced by the same point: (12) V1→=p1p2→=(x2−x1,y2−y1,z2−z1) (13) V2→=p1p3→=(x3−x1,y3−y1,z3−z1) (14) n→=V1→×V2→

After Equation (14) obtains the plane normal vector n→ , the relative positional relationship between the global plane and the local plane can then be expressed by the angle between its normal vectors n1→ and n2→ : (15) cosθ=n1⋅n2| n1 |×| n2 |

Equation (15) calculates the cosine angles of the two plane normal vectors n1→ and n2→ , and the final θ is obtained by the inverse cosine function, reflecting the relative positional relationship between the two planes.

The GL-Compound method was used after analyzing the dance posture changes, using global parameters combined with local parameters to represent the movement of each body region, and in this section, the similarity calculation of the posture between learners and professional dancers will be defined based on the dance.

Each beat movement contains five body torso and one core region movement parameters, the data dimension of GMP parameter is up to 4, the data dimension of LMP parameter is up to 3, and some body parts need not to be considered can be replaced by 0, so that the 7-dimensional movement parameters of each beat movement are established. According to different flying basic movements, the beats set up are also different, assuming that there are n beats of the dance movement being learned, then there are n samples of this set of dance movement data, each sample is a 7-dimensional vector, so that each set of flying basic movements is represented as an n × 7 matrix. The history of scoring dance movements cares about the coordination and proportion of movements between the whole body and parts of the learner, and does not go overboard with the length characteristics between body joints, so the use of cosine similarity is considered to calculate the degree of similarity of the movement parameter vectors of the corresponding body parts of the two sets of movements in the same beat. During the assessment process, in order to ensure that the student’s movements are synchronized with the template movements and to allow for certain time differences, a metronome interface needs to be designed with voice and text prompts for movement details, requiring the user to complete a specific movement within a given time frame, synchronizing the movement tempo as much as possible: (16) si=cosθ=a→⋅b→| a→|×| b→|

Equation (16) illustrates that when learners follow a dance movement, the feature vector of a body part used to represent the movement parameters is a→ , and the feature vector of the movement parameters of that part in the standard proto-language movement is b. The closer the cosine similarity s_i is to 1 indicates that the movement states of the two same body parts in space tend to be similar. Because different dance performance styles have different movement difficulties for different body parts and the importance they emphasize, it is necessary to go to consider the importance of each body part of Dunhuang Feitian Dance and the priority of the evaluation, set different weights Mi for each pair of body part feature parameter similarity (0.1 for the upper body, 0.25 for the left arm, 0.25 for the right hand buttock, 0.2 for the left leg, 0.2 for the right leg), and weight the obtained similarity weights are summed and averaged with n as the denominator, determined as the current complete one coherent action similarity score, as shown in Equation (17): (17) s=1n∑i=1nMisi

When the overall similarity S < 0.5 it is considered that the learner is not learning well. When the overall similarity is S>0.5, it is considered that the learner’s movements are standard. When the overall similarity is S>0.8, the completion is considered to be high and excellent: point out to the user the body parts that are less similar and emphasize the exercises to focus on.

Before the proposed detection method is put into the actual dance posture detection work, the feasibility and detection effect of the method need to be objectively analyzed and confirmed to be feasible before being put into practical use. The NTU-RGB+D human dance posture movement dataset was selected as the basis for this experimental study. The dataset is comprised of 60 human dance movement samples with diverse postures and dance types, resulting in a total of 56880 movement samples. The ballet dance gesture action video of the dataset is selected and the video is segmented, and the segmentation result is shown in Table 1, which is segmented into six segments of different actions.

In order to more intuitively verify the detection effect of the human dance posture detection method proposed in this paper, the human dance posture detection method proposed in this paper is set as an experimental group, and the detection method based on the improved ViT and the detection method based on the OpenPose model are set as the control group 1 and the control group 2, respectively, to carry out the comparative experimental analysis. In this experimental test, the intersection and merger ratio index is selected to evaluate the effect of human dance posture detection. The intersection and union ratio (IoU) is a criterion that can measure the accuracy of object detection within a specific dataset, which can accurately describe the similarity between the target detection area and the actual area, and the computational expression is: Iiou=(A∩U)/(A∪U)

Where:A denotes the dance posture movement interval determined by the detection algorithm.U denotes the actual detection of dance posture movement interval. When the intersection ratio is within the interval of 0.5~1.0, the human dance posture movement detection is considered accurate, and the closer to 1.0, the higher the accuracy. When the intersection ratio is less than 0.5, it is considered that the human dance posture movement detection is inaccurate. Six groups of dance posture movement segments were detected separately, and the MATLAB simulation software was used to determine and calculate the intersection and concurrence ratio values after the application of the three detection methods, and finally, the results were compared, as shown in Figure 3.

Figure 3.

The comparison of the three methods is compared to the index

There is a significant variability in the detection results after applying the three detection methods. In the experimental group, the cross-combination ratio of the human dance posture-based detection method proposed in the article is always in the range of 0.5~1.0, and all of them are over 0.85 and close to 1.0. After applying the method of control group 1, the cross-combination ratios of the dance posture segments 2 and 4 are less than 0.5, and the cross-combination ratios of the dance posture segments 3 and 6 are less than 0.5 after applying the method of control group 2. It is evident that the proposed dance posture detection method is highly feasible, and the detection accuracy advantage is significant. Proposed dance posture detection method is highly feasible and the advantage of detection accuracy is significant.

From the NTU-RGB+D dataset, different parts of the upper body, lower body, and the whole body regions are selected to extract RGB features, SIFT features, Flow features, and the recognition rate after combining SIFT and optical flow features for comparison. Table 2 displays the recognition results of different features extracted from different human body regions.

From the recognition rates obtained by different methods, it can be seen that the recognition rates of actions after feature extraction using the upper body or lower body are closer, but the recognition rate of the lower body is slightly higher (RGB, SIFT, Flow, and SIFT=Flow: 10.5, 26.2, 34.7, and 48.0), and the recognition rate is highest in the case of using a combination of human body regions, and in the case of the human body full-body region being recognized directly, the recognition rate is is lower than the recognition rate for the combination of body regions. The reason for this is that the number of dances with mainly upper or lower body movements is close to the number of dances with mainly lower body movements when selecting the movement category, and the number of dances with mainly lower body movements is high. In the application of the whole body region, when the movement is only upper body region or lower body region, the two regions will affect each other, and the naming of the dance movement often uses the lower body and upper body movements respectively, so the separate training can improve the accuracy of the recognition.

The experiment was done on a PC with Core(TM)i5-34703.2GHz CPU, 4GB RAM, and Matlab was used as the development environment. The motion database created contains 18 sets of dance movement clips, each set of dance movements of about 1200 frames, the experimental subjects are randomly selected college students, the experimental subjects have a basic knowledge of dance.

The experimental subjects were asked to imitate the standard movements in the database and make the corresponding dance movements under the motion capture system, to extract the joint point movement characteristics of the left arm of the experimental subjects and to compare them with the standard movements, and this paper mainly focuses on the final movements of the left arm in a single section of the dance movements (within 0-10s) for experimental comparison. Taking a segment of local motion sequence captured in real time as an example, the results are shown in Fig. 4, which shows the difference comparison between each main movement change of the subject to be tested and the standard movement. It is concluded from the difference comparison analysis that the degree of elbow bending of the subject to be tested in the 4-6s time interval and the amplitude of the left arm swing in the 6-8s time interval are significantly different from the standard action, and the difference comparison diagram of the left arm movement posture is given, from which it is obvious that there is a difference between the action to be tested and the standard action. Through the comparison of the experimental results, it is verified that the method of using GL-Compound similarity calculation and matching for the analysis of the movement posture can clearly and efficiently detect the differences and standardization between the movement objects, with high robustness, which lays the foundation for the scientific training of dance.

Figure 4.

Timing diagram of the left arm motion parameters

The problem of movement quality assessment, which is the assessment of the level of performance of a player in accomplishing a given task, is of great significance in the fields of sports, medicine, and teaching. For example, in a sport, the performance of dancers is given by experts who have been trained for many years in the respective field based on the athletic performance of the players, but without the on-site evaluation by these experts, the dancers would not be able to get feedback to improve their performance. Therefore, there will be a need to create an automatic assessment tool to automatically assess the quality of the movements performed by the dancers.

In this paper, the GL-Compound method is used to represent the movement of each region of the body using global parameters in combination with local parameters, and the scoring results are shown in Fig. 5, where the line graphs show the attention of the high-quality score features of the branches of the dynamic flow of a sample, and for the convenience of the presentation only 32 of the 256 segments of the attention are selected for the plots, which show the two high-attention segments and two low attention segments, the length of the segments is 16 frames, in order to facilitate the display only 8 frames are shown in the figure.

Figure 5.

Qualitative research on mass fraction decoupling module

According to the model design of this thesis, when a clip has a high similarity score of dance movements, it means that the clip shows movements that have a positive effect on the quality score, e.g., the contestant in the image clip from 50 to 100 frames completed a good Freeze movement with perfect body control, and thus received a high score, up to 0.94. The contestant in the image clip from 120 to 150 frames did not complete the movement, and their body posture was not perfect. movement was not completed and the control of the body posture was poor, thus receiving a lower score, as low as 0.17, and the clip represented by 225~250 in which the contestant only performed a turn, thus receiving a lower score for the calculation of the similarity of the dance movement.