Research on educational resource recommendation system based on MRLG Rec

Reinforcement learning is one of the main methods to solve most system algorithms, which can carry out iterative analysis and make exploratory evaluation of problems. Among them, the Markov decision process (MDP) can match the theoretical level of RL to create modular questions. The MDP is generally defined as A quad, that is, M = < S, A, R, P >, where S is the set of states, A is the action set, R is the reward function, R (s, a) refers to the reward value obtained by taking action A in state S and P is the state transition function, P (s, a, s′) is the probability of states transferring from a serialised action a to state s′. The root of RL is the strategy of maximising cumulative rewards through calculating learning degrees, which can be used by system core analysis to make subsequent decisions.

h¯: S→A indicates the instantaneous direction selection of the action in a certain state. For example, a = h¯ (s) indicates that a is selected in the instantaneous direction in the state S. Random strategy h: S×A→ [0, 1] represents the probability of selecting a single action in a certain state. For example, P (a ∣ s) = h (s, a) represents the probability of selecting action a in state s. The following text will define h to represent a policy so that it is easier to understand. Set instantaneous direction time as k, state as s_K, strategy as h, artificial intelligence according to s_k and strategy h select action a_K and immediate reward R (s_k, a_k), and the system will record this state as s_K +1. The operation theory of RL can be simplified the system shown in Figure 2.

Fig. 2

By above can lenovo education resource recommendation system based on RL is a one-way design: the initial state is equivalent to the learner (user) requirements, through RL deep cycle probability calculation and recording of MDP model, can be the user’s resources on keywords of radiation spread (action), leads to the results of multiple resources feedback. Then, according to the advantages and disadvantages of multiple resource results, the optimal result is obtained so as to meet the resource response of user groups. The results obtained in the circular radiation distribution of RL can be used as log records to integrate a new database to record the source point of teaching resources and simplify the generation time of subsequential repeated actions.

The GAN is an element calculation problem dealing with minimal and maximal boundaries. Its target value can be expressed numerically by the following Eq. (1): 1 minG maxD V(D,G)=Ex∼pdata(x)[log D(x)]+Ez∼pe(z)[log (1−D(G(z)))]

To explore the distribution p data of real data X, we can first set the noise variable p_z (z) and derive the mapping G(z; θ_g), in which case g is a differentiable function and the parameter θ_g represents the multilayer perceptron parameter. Meanwhile, the second multilayer perceptron is D(x; θ_d), whose output is a scalar. D(x) represents the probability that x comes from real data distribution. According to the training, the addition of model D can comprehensively improve the identification of whether sample sources belong to the generator model G. At the beginning of learning, the objective function V(D,G) cannot provide appropriate gradients for the generator model G for learning due to the small or even no reference capacity samples. When the generating capacity of the generator model G is insufficient, the generated sample can be intrinsically different from the real sample, and the discriminator model D can discriminate the authenticity of the output sample, and lg(1-D(G(z)) tends to 1. The subsequent generation capacity of the generator model G is enhanced. The smaller the parameter LG (1-D (G(z))) value, the larger lg(D(G(z)) value, and the final objective function V(D,G) will guide G and D at the same fixed point. The GAN theory can be simplified into the system shown in Figure 3.

Fig. 3

The continuous interaction between real data and the generated model, the internalisation of the resulting data records and the filtering, and the iterative process can continuously improve the performance of the generator model. In view of this, RL and GAN jointly form the recommendation algorithm of RL and GAN (MRLG Rec), and the running process is shown in Figure 4.

Fig. 4

As can be seen from Figure 4, the operation time required by the parallel operation of GAN and RL agent will be lengthened to some extent. However, the lengthening of time dimension can undoubtedly promote the organic learning of system self-consciousness. Similarly, incorporating the MRLG Rec algorithm into the educational resource recommendation system on the basis of the previous section will have more advantages than the single RL algorithm.

User base: The user database is used to store and write the characteristic information of the user group, including the nominal personality and behaviour personality. These characteristics are inherent to user groups and cannot be easily changed or altered. Personalised information includes but not limited to age, occupation, residential address and other early identification information used to confirm the user’s portrait. This information can be defined as static and does not change frequently. On the other hand, behavioural information is dynamic information, including user login address, click number and comment behaviour. Both static information and dynamic information are important elements of the educational resource recommendation system.

Repository: The resource library covers educational materials, learning materials and markers. In this database, theorems and algorithms in each chapter are regarded as branches of the knowledge tree. Learning resources can be derived from the branch concept nouns of the knowledge tree to match the corresponding resource information in the network space. As long as the source of the branch concept is searched and captured on the network, the teaching resource can be obtained.

User analysis: Combined with the collection recognition of user characteristic information in the data layer, the single educational resource recommendation system analyses the resource preference behaviour of user groups, then describes the user group portrait and classifies the group image. Therefore, this analysis is the basis for the educational resource recommendation system. Multiple users or user groups with high correlation will be designated as ‘neighbourhood’. On the basis of ‘neighbourhood’ users, relevant resources can be matched according to the correlation between current users and ‘neighbourhood’ users so as to achieve accurate and simple recommendation effect.

Resource analysis: Resource analysis integrates factors such as tags, clicks and resource comments as attributes of learning materials. The system builds a special database for the attributes and characteristics of learning materials and makes statistical modelling of this kind of information set so as to evaluate the similarity and quality of learning materials. The similarity analysis between learning materials is based on the modelling of learning resource recommendation. Echoing the concept mentioned in the user analysis, similar data can be obtained as ‘neighbourhood’ data. Once the system algorithm detects that a user is searching for similar data, the ‘neighbourhood’ data can be recommended to the current user. The quality analysis of learning materials is based on the evaluation of attributes such as post-use comments and usage of user data so that inferior educational resource content can be filtered accordingly.

According to the overlapping information collection of data layer and data analysis layer, a search recommendation mechanism can be formed by initiating search from user preferences, behaviour data and data resource collection radiation. In the above combination of forms, the multiple recommendation mechanism of the single educational recommendation system without strengthening the selection algorithm is shown in Figure 5.

Fig. 5

In order to measure the advantages and disadvantages of MRLG Rec educational resource recommendation strategy, the concept of state value function is introduced into the reinforcement algorithm, and the value function is used to evaluate the strategy. Specifically, the value function is divided into state value function V^h (s), and action value function Q^h (s, a). Where, V^h (s) is the accumulative expected reward that can be obtained according to strategy h in the pre-state s, while Q^h (s, a) is the accumulative expected reward that can be obtained according to strategy H in the current state action pair (s, a). V^h (s) and Q^h (s, a) can be considered fixed point solutions of the corresponding Bellman equation, which can be expressed as follows: 2 Vh(s)=∑a∈Ah(s,a)[R(s,a)+γ∑s′∈SP(s,a,s′)Vh(s′)] 3 Qh(s,a)=R(s,a)+γ∑s′∈SP(s,a,s′)∑a′∈Ah(s′,a′)Qh(s′,a′)

Here, γ is the discount factor. The optimal strategy h * refers to the strategy that can obtain the maximum cumulative reward, and its corresponding optimal value function V* (s) and Q* (s, a) can be expressed as follows: 4 V∗(s)=maxa∈A⁡{R(s,a)+γ∑s′∈SP(s,a,s′)V∗(s′)} 5 Q∗(s,a)=R(s,a)+γ∑s′∈SP(s,a,s′){maxa′∈A⁡Q∗(s′,a′)}

The purpose of the GAN is to improve the generative capability of the generator model G and discriminant model D so as to achieve the Nash equilibrium. Definition of intensive study in the process of learning sample set based on real experience (s, a), (after the state s′, r) reward come in pairs, on a moment s corresponding to the corresponding state of action a, (s, a) is known as the state of dynamic, migration to the state s′ of the next moment, and get an immediate reward r, (s′, r) is referred to as a follow-up state reward pair. Therefore, V*(s) and Q*(s,a) values can be substituted into the real empirical sample set Dx so that Dx = [s, a, s′, r] can be divided into two parts: 6 Dx=[(s,a),(s′,r)]=[x1,x2]

Here, x₁ represents the state action pair and x₂ represents the continued state reward pair. Since the subsequent state s′ and reward r are based on the state S at the previous moment and the corresponding action A, there is a definite connection between x₁ and x₂, and mutual information I is used to represent the association between the two: 7

Here, H(x₂) represents the entropy of x₂ and is used to measure the uncertainty of x₂. H of x₂ | x₁ is the uncertainty of x₂ given x₁. I(x₂.x₁) shows the reduction in the uncertainty of x₂ caused by x₁. Since x₁ and x₂ are correlated, mutual information I cannot be 0. Therefore, r-RU (relational correction unit) of the depth spiritual meridian network is constructed by using this method. The input of R-Ru is x₁, and the output is x₂. This relational correction unit is used to train the internal connection between X₁ and x₂ in the experience sample set.

Consistent with the experience sample set, the experience sample Gz=[sz,az,sz′,rz] generated by the GAN can also be divided into two corresponding parts: 8 G(z)=[(sZ,az),(sz′,rz)]=[G1(z),G2(z)]

Here, G₁(z) represents the generated state action pairs and G₂(z) represents the generated subsequent state reward pairs. In order to improve the quality of the generated samples, on the basis of the generated G(z), the generated G₂(z) and G₁(z) should conform to the structural relations in the real samples [x₁, x₂]. Therefore, by combining the generated sample G₁(z) and mutual information I, G₁(z) is input into the correlation correction unit R-RU, and the output of R-Ru is regarded as the subsequent state reward pair G₂(z) of construction. The goal is to make the generated subsequent state reward pair G₂(z) have a high similarity with the constructed subsequent state reward pair G₂(z)′. Relative entropy (KL divergence) is used to express the similarity between G₂(z) and G₂(z)′, and its equation is as follows: 9 DKL(P∥Q)=∑ip(i)log⁡1q(i)−∑ip(i)log⁡1p(i)=∑ip(i)log⁡p(i)q(i)

Definition = y₁ is the similarity accuracy of MRLG Rec educational resource recommendation system.

In order to measure the advantages and disadvantages of a single educational resource recommendation strategy, the behaviour sequence of users is defined as a finite set S: 10 {(z1,y1),(z2,y2),…,(zn,yn)},n≥2

Here, (z_i,y_i) represents the i-th element pair, z_i represents the access module and yi represents the corresponding operation, which is recorded in the set according to the sequence of behaviour occurrence. To simplify the description, the string (z_i,y_i) composed of elements in the element pair (z_i,y_i) is represented by Si, which is called the i-th state string of the user.

A sequence of states is a string of successively linked elements in each pair of elements in a sequence of actions. For example, the state sequence of the user S can be expressed as ‘S1,S2…Sn′, denoted as S=s₁,s₂…S_n.

The state subsequence of state sequence S is defined as follows: 11 S(i)=Sn1,Sn2,…Sni,1<n1<n2<…ni<s

Let the state sequence of user A and user B be A and B, respectively, and then the similarity of behaviour sequence is given by the following equation: 12 sim=α∗simseq(A,B)+β∗simtrans(A,B)+γ∗simvalue(A,B)

Among them, 13 α+β+γ=1,α≥0,β≥0,γ≥0

Represents state value similarity, state transition similarity and state order similarity respectively.

Based on the three similarity values, the learning behaviour of users in different time periods has different contributions to the prediction of their learning behaviour. Generally speaking, the closer the behaviour, the more it reflects the user’s interest in learning and contributes more to the similarity between users. In order to improve the importance of recent behaviour sequences to similarity calculation, time weight function WT was introduced.

14 WT(A,S)=(1−a)+aDA,SLA

Here, SA is the set of all the behaviour sequences of user A, and DA and Si represent the time interval between the behaviour sequence generated by user A and a behaviour sequence generated at the earliest. LA represents the time span of user A’s behaviour sequence and α ∈ (0, 1) is the weighted growth index. The calculation of user similarity between users A and B based on time attenuation effect is as follows: 15 sim(A,B)=∑Si∈SA[(W(A,Si)+W(B,Sj))/2]∗sim(Si,Sj)|SA|∗|SB|

When analysing relationships between users, it is not enough to consider only behavioural similarity. There are many reasons for the high degree of similarity, such as the inability to observe long-term differences in user behaviour over a short period of time. However, in practical application, a more accurate and stable description of the relationship between users is needed. Therefore, this article proposes the concept of correlation coefficient, that is, the similarity between users in a certain period of time can be obtained by analysing the change of similarity. Assuming that the average similarity is sim_avg and the variance is sim_dx, the correlation coefficient (RC) can be calculated by using the following equation: 16 RC=simavgsimdx

Therefore, the closer the relationship between two users, the greater the change of the average similarity. On the contrary, the change of average similarity is smaller.

Definition similarity accuracy of y₂ as a single educational resource recommendation system

Since energy consumption data are a time-series, the prediction model sample is also time-series structure. According to the previous analysis, the prediction model sample includes the following parts:

X = {(t₁,t₂,…,t_i),(t₂,t₃,…,t_i+1),…,(t₂,t₃,…,t_i+1)}, It is called the training sample set, which contains k energy consumption data of the first I moments.

It is called the reward sample set, and the reward value is the negative value of the absolute value of the difference between the action value taken in each state and the real value of the energy consumption at the next moment. The sample set contains K reward values and corresponds to each training sample in the training sample set. The ultimate goal of the algorithm is to maximise the cumulative reward.

In order to test the prediction performance of the energy consumption prediction model, the average absolute percentage error (MAPE) is used to measure the prediction accuracy. MAPE is the proportion between the error of the predicted value and the actual value, and its calculation method is shown in Eq. (17): 17 MAPE=1k∑i=1k|yi−yi′|yi⋅100% where k is the total number of samples used to evaluate model performance, y_i is the real energy consumption value and yi′ is the predicted energy consumption value. Therefore, the calculation process of the average accuracy of the energy consumption prediction model is shown in Eq. (18): 18 Y=1−MAPE=1−1k∑i=1k|yi−yi′|yi⋅100%

In order to verify the relationship between the interval m of the action space of the two recommendation algorithms and the search efficiency, the interval 0.5, 1, 2 of the action space is taken to compare the performance of the number of feedback resources of the two algorithms under different action intervals. The calculation results are shown in Table 1 and Figure 6. The data in the table represent the number of resources searched by the MRLG Rec educational resource recommendation algorithm and single educational resource recommendation algorithm in the same action space interval. During the experiment, each algorithm was independently executed 10 times, and the average value was calculated as the experimental result so as to reduce the occurrence of accidental situations. As can be seen from Table 1 and Figure 6, the interval M of action space is 0.5, 1 and 2, and the number of resources searched by the MRLG Rec educational resource recommendation algorithm is 4.85, 4.98 and 5.13. The number of resources searched by the single educational resource recommendation algorithm is 3.66, 3.75 and 3.84. In conclusion, in the same space interval, MRLG Rec search resource of the education resources recommendation algorithm is superior to the single education resources recommendation algorithm, and with the passage of action space interval, the number of search two algorithm has been increasing, widening of the gap is so MRLG Rec search advantage of the education resources recommendation algorithm is better.

Fig. 6

In order to verify the similarity accuracy index of the two algorithms for searching resources, the number of energy consumption contained in each state, n, was set to 2, 3, 4 and 5 to measure the percentage error of the corresponding similarity accuracy, y₁ and y₂. During the experiment, each algorithm was independently executed 10 times, and the average value was calculated as the experimental result. The experimental results are shown in Table 2 and Figure 7. The data in the table represent the error between the actual value of state energy consumption and the predicted value. As can be seen from Table 2 and Figure 7, when the amount of energy consumption contained in each state n is set to 2, 3, 4 and 5 the similarity accuracy percentage error of the MRLG Rec educational resource recommendation algorithm is relatively small, while the similarity accuracy percentage error of single educational resource recommendation algorithm is relatively large. To sum up, the MRLG Rec educational resource recommendation algorithm has higher accuracy of resource results, and the corresponding captured teaching resources are of better quality.

Fig. 7