Research on the Optimization of Personalized Learning Paths and Teaching Practice Strategies of Deep Enhanced Learning for Dance Choreographers

The deep DQN algorithm is a classic value-based deep reinforcement learning algorithm. DQN combines convolutional neural network with Q-learning algorithm in traditional reinforcement learning, and uses an empirical playback mechanism to store the transfer sample (s_t, a_t, r_t, s_t+l) generated by moving the machine into the environment interaction within each time step t into the playback memory unit D, wherein s_t represents the machine into the current state, a_t represents the action that the machine may produce in the current state, r_t represents the timely reward, and represents the immediate reward return obtained by the machine when it executes action a in the current state s, and r, s_t+l represents in s_t influences under the state of the next moment. In the training phase, a small number of batch transfer samples are randomly selected from the dataset D at certain intervals, and the stochastic gradient descent algorithm is subsequently used to iteratively learn from this batch of samples in order to adjust the settings of the parameters θ of the network.

The DQN model uses a deep convolutional neural network to approximate the optimal action-value function (7), where θ_t is the initial network parameters: (7)Q(s,a,θt) ≈ Q*(s,a) = maxπ E[rt+γrt+1+γ2rt+2+⋯|st=s,at=a,π]

DQN In addition to using a deep convolutional network to approximate the current value function, the DQN model uses another network Q(s, a, θ_t−) separately to generate the target Q values, where θ_t− is the updated network parameters. That is, Q(s, a, θ_t) is the output of the estimation network and Q(s, a, θ_t−) is the output of the target network. The parameters in the estimation network are updated in real time. After a number of steps, the parameters of the estimation network are replicated in the heliograph network and then the target network remains unchanged and the estimation network continues to be updated in real time. The update of the DQN network parameters is done by utilizing the TD error (temporal differencing) with the following equation: (8)L(θt)=Es,a,r,s′[(Yt−Q(s,a|θt))2]

At network initialization θ_t = θ_t, where Y_t is an approximate representation of the optimization objective of the value function: (9)Yt=r+γ maxat+1Q(st+1,at+1|θt−)

The gradient formula is obtained by using SGD to obtain the partial derivatives for parameter θ: (10)L(θt)=Es,a,r,s′[(Yt−Q(s,a|θt)∇θtQ(s,a|θt)]

A significant drawback of the DQN algorithm is the Q-value overestimation problem, where the intelligent body selects the action with the highest Q-value based on a greedy strategy, when the action is not necessarily optimal [28].

Policy-based methods are suitable for continuous or high-dimensional action spaces and have the advantages of simple policy parameterization and fast convergence.

The REINFORCE algorithm is a typical policy-based deep reinforcement learning algorithm. It is based on the idea of gradient ascent to maximize long-term returns by directly updating the parameters of the policy function. The core idea of the REINFORCE algorithm is to use the policy function to define the probability distribution of choosing an action in a given state, and then compute the gradient based on the trajectories obtained from sampling, and ultimately use the gradient ascent method to update the parameters of the policy function.

The advantage of the REINFORCE algorithm is that it is able to self-adjointly optimize the policy function without the need to estimate the value function, making it suitable for problems in both discrete and continuous action spaces. However, the REINFORCE algorithm also has some disadvantages, such as low sampling efficiency and high variance [29].

The Actor-Critic algorithm is a reinforcement learning algorithm that combines a policy gradient approach and a value function approach. It consists of two parts: 1)

Actor is responsible for tuning the parameters θ of strategy π_θ(a ∣ s).

Actor network can be described as a network that finds the probability of all available actions and selects the one with the highest output value, while Critic network can be described as a network that evaluates the selected action by estimating the value of the new state resulting from the execution of the action [30].

The Deterministic Policy Gradient (DPG) algorithm is a common Actor Critic algorithm, and the DPG models policies as deterministic policies μ(s). Deterministic policies are a special case of stochastic policies, where the objective function of the target policy is averaged over the distribution of states of the behavioral policy: (11)J(μθ) = ∫sρμ(s)∫Aπθ(s,a)r(s,a)dads = ∫sρμ(s)r(s,μθ(s))ds = Es−ρμ[r(s,μθ(s))]

Compared to the stochastic strategy, the deterministic strategy gradient removes the integral over the action, the gradient only integrates over the state, so there is no need to sample the importance of the action, then the gradient becomes Eq. (12), which greatly improves the efficiency: (12)∇θJ(μθ)=∫sρμ(s)∇θQμ(s,μθ(s))ds =∫sρμ(s)∇θμθ(s)∇aQμ(s,a)|a=μθ(s)ds =Es−ρμ[∇θμθ(s)∇aQμ(s,a)|a=μθ(s)]

In this study, learning target features and domain knowledge features are added to characterize the dynamic learning environment as State = [e_t, Target, p(K^t), p′(K^t), Dif₀], State_t specific descriptions are: e_t denotes the current learning object, Target is the target knowledge point concept, the learning target can be formulated by the teacher or freely decided by the learner according to his/her own situation before the start of the learning, p(K^t) is the cognitive state of the learner at the moment of t, p′(Kⁿ) is the predicted next step of the knowledge point at the moment of t concept coverage, Dif_t is the difficulty value of the learning target.

In this study, the LSTM model was used to predict the cognitive state of the learner, the Transformer model was used to predict the conceptual coverage of the next knowledge point, and the dynamic difficulty value of the learning object was calculated based on the cognitive state of the learner.

Suppose there is a C course containing a total of K knowledge point concept, denoted as K = {k₁, k₂, ⋯, k_|K|}. The learner is denoted as S = {s₁, s₂, s₃, ⋯, s_||}, the exercise bank is denoted as EB = {e₁(k), e₂(k), ⋯, e_lenl(k)}, and the exercise is denoted as e_j(k) = [e_j(k₁), e_j(k₂), e_j(k₃)⋯, e_j(k_|K|)], e_j(k_i) taking the value of 0 or 1 (0 means that the question does not contain the ith knowledge point concept, and 1 means that the question contains the ith knowledge point concept). The history of answer records of a learner s_i is denoted as X_si = {x₁, x₂, x₃, ⋯, x_|si|}, and the answer of learner s_i to exercise e_j at the tth moment is denoted as: (13)xt={(si,ej(k),at)|si∈S,ej(k)∈EB}

The ISTM model is used to predict learners’ mastery of knowledge concepts and to track their cognitive state. The input to the LSTM model is xt={(si,ej(k),at)|si∈S,ej(k)∈EB} , the single-hot encoding of the knowledge point concepts of Exercise e_j is denoted by ϕ(Kⁿ), and a^t takes the value of either 0 or 1 (with 0 and 1 denoting an incorrect and a correct answer, respectively). The output of the model h^t is a vector whose length is equal to the length of K. Each of its components represents the probability of answering the corresponding knowledge point concept correctly. In this study, this model is trained by constructing a loss function through binary cross-entropy, and the optimized loss function for a single learner is denoted as: (14)Ls=∑t=0Tlb(ht⋅ϕ(K1),at+1)

where · denotes dot product and l_b denotes binary cross-entropy loss.

When the training of the LSTM model is finished, the historical answer records of a learner are inputted and the output of the model is his mastery of all the knowledge point concepts of the course, denoted as p(K′)=[p(k1t),p(k2t),p(k3t)⋯,p(k|K|t)] .

The Transformer model is used to predict the knowledge point concept coverage to pinpoint the knowledge point concepts that the learner should learn next. The positional coding is embedded in the input of the Transformer model to characterize the sequential information in the historical learning record, and the input of the model is denoted as: (15)ε(t)=Eet,kt+Pt

where E_e, k is the embedding vector connecting E_t and K_t, and P_t denotes the positional embedding.

The Transformer model utilizes a decoder to predict the conceptual coverage of knowledge points for the next question. The decoder is connected to the encoder through a self-attentive mechanism and finally the output of the model is obtained through the full neural network v^t. At the training moment t, v′ denotes the probability of occurrence of all the knowledge point concepts in the course i.e., K_t+l. The model is obtained by minimizing the loss function L′. The loss function of the training model is denoted as: (16)Lss′=∑t=0Tlb(vt⋅ϕ(kt+1),1)

When the training of the Transformer model is finished and a learner’s record of exercises is input, the output of the model is the probability of occurrence of all the Knowledge Point concepts in the course, denoted as C(Kt)=[c(k1t),c(k2t),c(k3′),⋯,c(k|K|′)] . In order to discover new Knowledge Point concepts that have to be learnt, a weighting variable is added to the output of the Transformer model, which is equal in length to the length of the Knowledge Point concepts, denoted as ω(Kr)=[ω(k1t),ω(k2t),ω(k3t),⋯,ω(k|K|t] . The computation of ω(kit) as follows: (17)ω(kit)={ 1−rici ci>0 1 ci=0

where r_i is the number of times knowledge point concept k_i is answered correctly and c_i is the number of times k_i occurs. Utilizing p′(K′)=c(Ki′)ω(Ki′) , the final knowledge point concept coverage for the next question is p′(K)=[p′(k1′),p′(k2′),p′(k3′)⋯,p′(k|K|′] .

Difficulty is a core factor to be considered in the process of recommending study materials for choreography, and this study utilizes Equation (18) and Equation (19) to calculate the difficulty of the exercises: (18)Re(K)=∏i=1n(p(kit)|e(ki)=1)

R_e(K) is the probability of answering the exercise correctly, p(kit) is the degree of mastery of each knowledge concept in the exercise. t the moment exercise difficulty value Dif_t for: (19)Difr=1−Re(K)

The Actor-Critic components designed in this paper include: states, actions, and return values.

State: this study considers the dynamic learning environment as the state of Actor-Critic, characterized as State = [e_t, Target, p(K^ι), p′(K^ι), Dif_t].

Action: the strategy network is a pre-trained neural network model, which accepts the learning environment state State, samples from the action space C_t according to the already saved model parameters θ, predicts and outputs the exercises that have the highest fitness with the current learning environment, and finally updates the learning environment to Statet′ . In this study, we obtain the conceptual coverage of the knowledge points for the next prediction from the dynamic learning environment, and retrieve the relevant exercises from the exercise library to form a candidate. Relevant exercises are retrieved to form a candidate exercise set C_t, which is calculated as follows: (20)Ci={ ej(ki)|ej∈EB,ki∈p′(K′)}

where e_j(k_i) is the candidate exercises at moment t, j is the number of exercises, and p′(K′) is the knowledge concept coverage predicted above.

Reward value: this study refines the method of calculating the reward value by giving a certain reward at each step of the intelligent body’s exploration and at the end of the exploration, and the reward value function is designed as follows: (21){ RStatei=α*Rstepi+β*RCSEAL|α,β∈{0,1} RStepi=1−|δ−Difnt|

R_CSEAL is the design function for the reward value, and RStepi is the reward value given at each step. In the early stage of the intelligent body’s exploration, this study sets α = 1, β = 0, then RStatei=RStepi , which represents the reward value given to the intelligent body at each step of exploration. Where δ is the difficulty of the exercise desired by the learner, the smaller the value of |δ − Dif_t|, the exercise is the one that best meets the needs of the learner. When the intelligent body completes the exploration, this study sets α = 0, β = 1, then RSatei=RCSEAL , which represents the return value of the whole adaptive learning path obtained by the intelligent body to reach the concept of the target knowledge point.

The D3ON algorithm is used to realize the choreography study material recommendation function. Two Q networks are set up as participants, i.e., the evaluation network Q is used to obtain the exercise corresponding to the maximum return value in state State_t+1, and then the target network Q is used to calculate the real return value of the exercise, so as to obtain the target value. The interaction of the two networks effectively avoids the problem of “overestimation” of the algorithm. Where θ and θ represent the parameters of the evaluation network and the target network, respectively. The target value is calculated as follows: (22)yt=Rt+1+γQ′(Statet+1,argmaxaQ(Statet+1,a,θ),θ′)

Where, argmax_aQ(State_i+1, a, θ) denotes the State_i+1-state evaluation network Q selects the exercise with the largest payoff value based on its parameters θ, and the action selected for this exercise is again computed by the target network Q to obtain the final true payoff value y_to On the basis of the computed y_t, the mean squared error loss function is used, the Loss is computed, and the parameters are updated by back-propagation θ. The formulae are as follows: (23)Loss=(1m)∑t=1n(yt−Q(Statet,Ct,θ))2

After the algorithm is run for many iterations, the policy network is trained. When all the variables in the dynamic learning environment model constructed above are input into the neural network, the corresponding exercises can be output.

Considering the characteristics of personalized learning path recommendation and the current research status, this paper adopts the “Dance Choreography” learning platform constructed with JSP+MySQL technology as the experimental object, and analyzes the experimental effect of the constructed personalized learning path recommendation model.

The “Dance Choreography” e-learning platform consists of four modules, namely, learning navigation, learning resources, problem solving and exploration, and learning interaction module. The knowledge items in the modules are categorized according to the chapters of the knowledge points. Among them, the resource navigation module consists of learning objectives, knowledge tree, key points and difficulties. The learning resources module consists of videos, e-lessons, and textbooks. The Problem Solving and Inquiry module consists of example problem analysis, exercises, and quizzes. The learning interaction module consists of discussion forums.

In order to improve the system’s extraction and programming of learning user access paths, it is necessary to further optimize the processing of the original log data, firstly, the knowledge items under the learning module are redefined, as shown in Table 1.

In this paper, a web page data collector was used to capture the learning data of 80 users in the log of the web platform. Considering the limitation of the learning materials of dance choreography on the scope of learning modules selected by users, the experiment chooses “Practice of Dance Choreography”, which has a comprehensive distribution of learning modules, as the experimental collection area. The number of visits to learning user nodes, learning paths, and test scores are obtained. Among them, the learning user node access volume refers to the click volume and time duration of the learning user for each knowledge item node, as shown in Table 2.

Based on the similar learning user model building method, 8 groups of similar user clusters were established for 80 learning users, and the recommended TopN-1 was calculated. The parameters were initialized and calculated according to the ant colony algorithm, which mainly included the user and learning style similarity value C_j, the heuristic information value η_ij obtained from the cognitive level of the learning user and the difficulty of the learning material d_j, and the optimization information value of the learning user’s evaluation, τ_ijnew, α pheromone factor, β heuristic function factor. Parameters are calculated as: η_ij = 0.5, τ_ijnew = 0.5, α = 3.5, β = 5. The calculation obtains the maximum probabilistic knowledge item recommendation TopN-2, and the personalized recommendation path is obtained after the ordered merger of TopN-1 and TopN-2, as shown in Table 3.

Starting from the target demand of personalized learning path recommendation, we introduce 2 performance indicators, learning efficiency and learning maze guide control effectiveness. Learning efficiency indicates the rate of improvement in learning performance after a period of continuous use of personalized learning path recommendation. Learning effectiveness is measured by the rate of increase of knowledge item check-in after learning users use the personalized learning path recommendation program compared with the previous one. The higher the concentration of knowledge item check-in, the higher the degree of the solution to the problem of learning lost in the process of students’ online learning.

To this end, a definition of knowledge quantity is first introduced: an e-learning platform is a network system composed of multiple knowledge nodes, each of which consists of n knowledge item. Knowledge nodes and knowledge items constitute a knowledge quantity, denoted as KI, where, for a certain knowledge point t the knowledge items that a student has checked in constitute a knowledge quantity of KI(t), and the personalized recommended knowledge items adopted by the learning user constitute a knowledge quantity of KI(t, s). Therefore, the effectiveness of the Lost Guide Control can be denoted as: (24)Ef(Ucur)=(∑t∈UcurKI(t))+∑t∈UcurKI(t,s)×UallKIall×Ucur

U_cur = (t₁ + t₂, ⋯, t_L) denotes the set of learned knowledge points, and U_all is the set of all knowledge points in the learning platform. KI_all represents the total amount of knowledge in the whole learning platform. As shown by the learning maze guide effectiveness formula, the more recommended knowledge items an individual acquires when learning a certain knowledge, the more effective the learning maze solution is.

Five dance choreography learning users are randomly selected from each of the eight groups of similar user groups for personalized learning path recommendation of dance choreography, and the lost-guide-control rate of personalized learning path recommendation is obtained respectively, as shown in Table 4.

Comparing the density of choreography knowledge item check-ins before and after personalized learning path recommendations, Figures 2 and 3 demonstrate the density of knowledge item check-ins before and after personalized learning path recommendations, respectively. Figure 4 shows the achievement trend after personalized learning path recommendation.

Figure 2.

The knowledge item check-in density before recommendation

Figure 3.

The knowledge item check-in density after recommendation

Figure 4.

The development trend of dance performance after recommendation

From the data, it can be seen that through the personalized learning path recommendation, the learning of choreography course learners have a certain degree of guidance and control, which is manifested in the acceptance of choreography recommended learning content recommendation, choreography learners check-in density is significantly higher than the recommended before the learning user check-in density. After receiving the path recommendation, the dance choreography performance of the dance choreography learning users are improved, especially for the dance choreography level between 60-70 points and 70-80 points of the learning users, their dance choreography level improved significantly.

The post-test data is the end-of-semester exam results at the end of the optimized and recommended teaching experiment of dance choreography learning path. The students chose the dance choreography test questions by drawing lots and completed the work choreography on the spot, and each student’s dance choreography exercise was scored and judged separately by the dance instructor of the preschool education major. The obtained post-test scores were obtained as shown in Table 5, and it was found that the average score of the experimental class was 0.54 points higher than that of the control class, and further T-test analysis was carried out to test whether the difference was statistically significant.

The data of the pre and post-test scores of the experimental class of dance choreography were entered into SPSS19.0 software for paired samples t-test respectively, and the results obtained are shown in Table 6. The obtained Sig value is 0.000, which is smaller than the critical value of 0.05, so the original hypothesis that there is no significant difference between the overall means represented by the two samples can be rejected. In other words, the paired-sample t-test results of the achievement data measured before and after the implementation of the personalized learning path recommendation-oriented dance choreography teaching experiment for the students in the experimental class show a significant difference, with the post-test data showing a more pronounced increase than the pre-test data.

From this, it can be judged that the experimental class students’ performance in the dance choreography course improved more significantly at the end of the experiment applying the new teaching strategy than before the experiment began.

The t-test results for each sub-item of the experimental class are shown in Table 7. The p-values obtained from the paired samples t-tests of the pre- and post-test scores of each sub-item of the students in the experimental class A were all less than the critical value of 0.05.

After the implementation of the personalized learning path recommendation-oriented choreography teaching experiment, the students’ choreography scores (post-test data) in the experimental class, including the three weaknesses of “structural design”, “movement choreography”, and “basic skills”, have increased significantly compared with the choreography scores before the experiment (pre-test data), by 11.68% and 8.54% respectively. After the implementation of the experimental dance choreography teaching experiment, the choreography performance of the students in the experimental class, including the three weak areas of “structural design”, “movement arrangement” and “basic skills” (the post-test data), has increased significantly compared with that of the choreography performance before the experiment (the pre-test data), with an increase of 11.68%, 8.54% and 16.69%, respectively. This also means that the students in the experimental class had their weak links in dance choreography effectively improved in this learning path recommended teaching experiment, and their dance choreography scores improved more significantly compared to those before the experiment started.

The data of the pre-test and post-test scores of the control class B students were entered into the SPSS19.0 software for paired samples t-test respectively. The results of the paired samples t-test for the control class students on the pre-test and post-test of the experiment are shown in Table 8.

The Sig value of the t-test for the overall mean score is 0.541>0.05, so the original hypothesis that there is no significant difference in the overall level of dance choreography before and after the experiment in the control class is not rejected, i.e., the students in the control class before and after accepting the traditional mode of teaching do not show statistical significance despite the fact that there is a slight increase in the performance of dance choreography. In the t-tests for each subdimension of choreography achievement, the t-statistics for all subdimensions were greater than 0.05, indicating that the differences in pre- and post-test scores for each subdimension were not significant.

It is inferred that the students’ dance choreography level did not achieve significant improvement after the control class was taught using traditional teaching strategies, and its teaching effect was not as effective as the recommended teaching strategy of personalized learning path optimization proposed in this paper.