Design and evaluation of intelligent teaching system on basic movements in PE

In order to obtain the coordinate information of key points in the human body, a Gaussian modelling method [9] is used to obtain the confidence diagram of key points, where the value represents the probability of being a certain key point. The confidence diagram of the key point position can be expressed as: 1 Cj,k=exp⁡(−∥p−xj,k∥22δ2) \begin{align}C_{j,k}=\exp{\left( -\frac{{\parallel{}p-x_{j,k}\parallel{}}_2^2}{{\delta{}}^2}\right)} \end{align} 2 Cj(p)=mSj,k(p) where j represents the joint point of the human body, k is represented as kth target person in the image, p ∈ R² represents the coordinates of the predicted person, x_j,k ∈ R² represents coordinates of the jth level of the kth target and δ is a minimum value, which can be used to ensure the feasibility of the model training. Eq. (3) indicates that when the predicted current position p is in the jth target character, the closer the key points are, the higher the score obtained. Eq. (4) indicates that k with the maximum score is found at the jth key point at the current position, that is, the target to which the key point belongs. The confidence diagram of limbs is used to represent the direction in the process of joint connection of limbs, which is similar to finding the starting point and ending point of a vector. For the kth target person, the PAFs of the cth class limbs such as forearm, trunk and thigh can be expressed as: 3 v=(xj2,k−xj1,k)∥xj2,k−xj1,k∥2 4 0≤v⋅(p−xj,k)≤lc,k where x_j,k represents the position of the j-class junction of the kth target person, and lc,k=∥xj2,k−xj1,k∥2 indicates the length of the limb part between two joint points.

OpenPose is the first open source library for estimation of real-time multi-person pose based on deep learning. It can accurately estimate the posture of everyone in the image in real time, and realise the extrmovement of face, trunk, limbs and hand bone points, and this representation is both real-time and accurate with strong robustness [10, 11]. The core of this method is a bottom-up human pose estimation algorithm based on partial affinity domain, that is, detecting key points before obtaining the skeleton, which avoids the problem of long calculation in multi-person scenarios.

2.2.2

Network structure of multi-level prediction

Figure 1 shows the network structure of multi-level prediction designed by OpenPose. The framework is based on VGG19 network model, and the input image is transformed into image features F, L(p) and S(p) by stage prediction. L(p) is the affinity vector field PAFs, and s(p) represents the confidence of key points in the skeleton. This structure divides the prediction into six stages; the first four stages predict the affinity vector field and the subsequent two stages predict the confidence. In each successive stage, the prediction result of the previous stage is connected with the original image features as input to generate a finer prediction. After obtaining the confidence and affinity of key points, the Hungarian algorithm is used to optimally match the adjacent key points, and the skeleton information of each person is obtained [14]. OpenPose has good real-time performance where a variety of model architectures are designed to be compatible with different hardware configurations. It uses a monocular camera to obtain reliable key information, without a camera containing special depth such as Kinect [15]. The estimated parts are eyes, ears, nose, neck, shoulder, elbow, wrist, hip, knee and ankle.

Fig. 1

In the field of movement analysis and recognition based on computer vision, the single tag classification method aims to solve the problem that an example belongs to only one category where different tags are completely independent and unrelated to each other. However, in practical application, a person’s limb movements may need to be analysed in many aspects, and there are often multiple tags in one frame of image, such as simultaneous analysis of upper limb and lower limb in one movement. In the problem of multi-label classification, labels are not completely independent, and there is a certain dependence or mutual exclusion [16]. In addition, the number of labels is relatively large in the multi-label classification task, which makes the relationship between categories complicated.

Fig. 2

Fig. 3

The methods of multi-label classification can be divided into two categories: problem conversion and algorithm adaptation [17, 18]. The method based on problem conversion usually transforms multi-label classification problems into other learning scenarios, such as transforming multi-label classification problems into multiple binary classification problems. The label sorting method is the most commonly used, and transforms the problem of multi-label classification into the problem of sorting multi-labels. If there are n labels, it is necessary to set n×(n−1)/2 label pairs; each label pair can be expressed as (y_j, y_k). In the construction of a binary classifier, samples belonging to tag y_j but not tag y_k are labelled as positive samples, and samples belonging to tag y_k but not y_j are labelled as negative samples. In the test process, when the final result is less than zero, the tag belongs to y_k, and when the result is greater than zero, the tag belongs to y_j; and then the result of classification by two classifiers is voted. In the voting process, a virtual tag y_v is added to divide whether there is a correlation between tags. Finally, not only the set of classes to which the samples belong but also the order of category tags according to the correlation degree between tags and samples is given.

Basic sports movements are the basic learning content of PE curriculum. As the foothold of basic sports movement teaching and improvement of students’ physical quality, it is an important content of developing sports activities in PE class. A large number of PE movements are designed through the combination of various basic movements and the arrangement of changing rhythm, whose movement route and sequence usually have their own specific requirements, and so beginners may have many problems in learning coherent movements [19], which are mainly the following:

Inconsistency of upper and lower limbs: For example, when students make basic movements such as standing still with their hands and feet, they often have the same hands with feet.

In order to evaluate the standard degree of movement through gesture matching in basic movement teaching of PE, a PE teaching system is designed with the basic process as follows. First, students learn from the template movement videos that are uploaded by teachers at the client, constantly practice the basic movements, imitate the standard movements of teachers in the videos as required and strive to achieve the specified movements. Then, students will upload and submit the video of movement learning results with their mobile phones, and wait for the systematic evaluation. After getting the video uploaded by the students, the system further processes the video.

The server is implemented by SSM framework and MySQL database [20, 21]. The SSM framework is a combination of three frameworks, namely Spring framework, SpringMVC and MyBatis; it has been widely used in the development of Web sites, and has become more and more popular in the development of commercial software. The APP client of this system adopts the current popular basic model of MVP[22] that consists of three major components: View, Model and Presenter. View is responsible for displaying pages, Model is responsible for providing data support for business processing and Presenter is responsible for processing business logic. The architecture of the system is shown in Figure 4. Users transmit data through cameras in mobile phone, MySql database is used to store data where the server accepts users’ data requests and the processed result is provided as feedback to the user.

Fig. 4

In the design of each functional module of the system, the relationship between each module and the calling mode is mainly designed. In order to meet the principle of high cohesion and low coupling in the process of software design [23], each module should be reasonably divided and each function should be described in detail. Through this process, not only should the basic requirements be met but also the maintainability and scalability of the software should be considered. The module of the PE teaching system is shown in Figure 5.

Fig. 5

It can be seen from the figure that the system is divided into seven modules: system login module (administrator login, common user login), model upload module, user management module, message release module, data upload module and analysis report viewing module.

In the teaching research of basic movements oriented to PE, the correctness of the students’ movements is evaluated with reference to teachers’ movements, and the similarity between their movements is calculated with the DTW algorithm based on the differences in pose characteristics mentioned above. The smaller the regular path distance between the two movements data sequences, the higher the similarity.

Motion pose feature sequence to be matched is expressed as P={p1,p2,…pi,…,pn} and template movement pose feature sequence as Q={q1,q2,…qj,…,qm} , where p_i and q_j are, respectively, the feature vectors of human differential pose, namely pi=(xi1,xi2,…,xi21) and qj=(yj1,yj2,…,yj21) . Euclidean distance is used to express the similarity between two features. The smaller the distance, the more similar it is, while the larger the distance, the smaller the similarity. Further, calculations can be made to determine the similarity of two frames of motion, whose similarity between the movement in frame i of the sequence to be matched and the movement in frame i of the template sequence is expressed as Eq. (8): 8 d(i,j)=121(xi1−yj1)2+(xi2−yj2)2+⋯+(xi21−yj21)2

In the ideal standard case, when the distance is 0, it means that the two features are exactly the same, and it can be judged that the two movements are exactly the same. Attitude feature information is standardised arc data, and the distance is less than 2π in extreme nonstandard cases. Regarding Euclidean distance as a similarity for DTW algorithm, matching can be a good way to search the best regular path, but it is difficult to quantitatively score different people who make the same movements. Therefore, the search condition of the maximum matching principle is redefined such that the Euclidean distance is scored and converted with the similarity converted to [0, 1], which means that the smaller the distance, the greater the similarity. The specific calculation is shown in Eq. (9): 9 sim(i,j)=11+d(i,j)

In Eq. (9), sim(i, j) = 1 when the ideal time distance d(i, j) = 0, whose hundred differentiation value is 100. It represents the movement in frame i of the sequence to be matched relative to the movement in frame s of the template sequence, whose score is expressed as Score (i, j): 10 score(i,j)=sim(i,j)×100

Replacing the original Euclidean distance in the matching process as the basis of similarity judgment with the movement score score(i, j), the implementation of DTW algorithm based on difference of pose feature is as follows:

Construct a matrix S with a size of n × m, and the matrix element s_ij = score(p_i, q_j) where the larger the value is, the more similar the movements in the two frames.

Based on the above algorithm, the movement that to be matched is marked as the movement with the highest matching score, that is, the most similar movement is used to identify its category, and the effectiveness of the teaching system is verified by the correct matching rate [24].

Taking broadcast gymnastics as the evaluation object, each movement of the official video is manually cut into a single segment as a movement template library, in which three movement segments are captured in every eight beats, the first four eight beats are used in each session, 108 standard movements are obtained in nine sessions as a template library and 150 samples to be matched for each movement are used for matching. The movement with the highest score is obtained as the matching result and the matching success rate is calculated. A high matching rate indicates a better matching effect. The results are shown in Table 1.

From the results, it can be seen that the successful matching rate is excellent for movements with less self-occlusion, which indicates that this algorithm can well eliminate the problem that the length of the movement frame sequence to be matched is inconsistent with that of the template movement frame sequence. The matching rate of simple slow movements in the plane, such as stretching movement, body movement and finishing movement, is high. However, when there are many areas of self-occlusion of limbs in kicking exercise and abdominal back exercise, the included angle of limbs, the vector of gravity of human body and the vector of gravity of limb block are lost, and the success rate of matching is perceptibly reduced.