Research on Machine Learning Program Generation Algorithm Based on AORBCO

This article uses the RippleNet algorithm to implement the structural information calculation module of Ego. The input to the algorithm is a user item pair, the output is the probability of a user clicking on an item. This article views the dataset object as a user in a machine learning knowledge graph, an algorithm object as an item, and the relationship between the dataset object and the algorithm object as a historical interaction record. On this basis, use RippleNet to implement the algorithmic decision-making ability of Ego. The calculation process of RippleNet is shown in Fig. 3.

Figure 3.

RippleNet calculates the interaction probability between two entities by searching for potential path information between user history click records and recommended items. The algorithm is executed as follows: a)

Model input: The model accepts users and items as inputs. User is represented by d, used to obtain the user's historical click history; The item is represented by m, representing the item to be predicted and clicked on.

For the implementation of the descriptive information computing module, this paper proposes the TCF (Text-based Collaborative Filtering) algorithm that makes improvements to the NCF (Neural Collaborative Filtering) algorithm [8], which uses the NeuMF to jointly train the GMF and MLP, and utilizes text data to achieve recommendations. For joint training and utilizes text data to achieve recommendations. The algorithm takes descriptive information about the user-item as input and the output is the probability of the user clicking on the item.

The overall structure of TCF is shown in Fig. 4 above. Specifically, using t to denote the initial input text and n to denote that there are n words in the initial input text, then the descriptive text corresponding to the object can be denoted by t = w_1:n = [w₁, w₂, …, w_n]. In this paper, we use GloVe [9] to initialize the embedding representation of each word w_i and obtain the sentence representation s by accumulating the representations of each word.

Figure 4.

After text embedding, this paper obtains the vector forms s_d and s_m of the dataset and the descriptive text of the algorithm. In order to dig deeper into the interaction information between the two feature vectors, this paper uses the GMF and MLP layers to analyze the linear and nonlinear correlations of the dataset and the algorithm descriptive features, and then fuses these two types of information using the NeuMF layer to calculate the interaction probability of the dataset and the algorithm.

The linear interaction between the dataset and the algorithm description features can be obtained through the GMF layer as shown in equation (5) and equation (6) below: (5) φ1=sd⊙sm {\varphi _1} = {s_d} \odot {s_m} (6) y1=σ(GTφ1) {y_1} = \sigma ({G^T}{\varphi _1})

Here, ⊙ denotes the product of elements. G^T is the weight matrix that can be obtained through learning. σ Denotes the Sigmoid activation function. This step can be interpreted as a special kind of matrix decomposition and has a higher expressive power than its original form.

While obtaining the linear interaction describing the features, this paper obtains the nonlinear interaction relationship between the two through the MLP layer. Specifically, this article connects two feature vectors and captures nonlinear interactions between features through a multi-layer fully connected network. The equation (7, 8 and 9) are as follows: (7) h0=sd∥sm {h_0} = {s_d}\parallel {s_m} (8) φnl=hn=σ(WnThn−1+bn) {\varphi _{nl}} = {h_n} = \sigma (W_n^T{h_{n - 1}} + {b_n}) (9) ynl=σ(WTφnl) {y_{nl}} = \sigma ({W^T}{\varphi_{nl}})

The symbol || represents the concatenation operation between feature vectors. In order to integrate linear and nonlinear interactions of text features, this paper connects the last layer of GMF and MLP, and fuses linear φ_l and nonlinear feature φ_nl through NeuMF layer to better learn implicit interactions between descriptive texts and predict the final interaction probability of d and m. Its equation (10) is as follows: (10) Yt=σ(WtT(φl∥φnl)) {Y_t} = \sigma (W_t^T({\varphi_l}\parallel {\varphi_{nl}}))

Finally, combining the two probabilities, equation (11) is as follows: (11) Y=WT(σ(dTm)+σ(WtT(φl∥φnl))) Y = {W^T}(\sigma ({d^T}m) + \sigma (W_t^T({\varphi_l}\parallel {\varphi_{nl}})))

Among them, σ(d^T m) calculates the interaction probability between objects based on their structural information, σ(WtT(φl∥φnl)) \sigma (W_t^T({\varphi_l}\parallel {\varphi_{nl}})) calculates the interaction probability between objects based on their descriptive information, and W^T is the weight matrix.

In this paper, the CodeT5 family of models is used as the basic model for code generation, on the basis of which further innovations are made to obtain the CodeT5+ model. First, the model introduces a flexible mode selection mechanism, which enables it to run flexibly in encoder-only, decoder-only, or encoder-decoder modes according to the needs of different tasks. This design makes CodeT5+ more adaptable to different types of downstream tasks and improves the generality of the model. Second, CodeT5+ employs a multi-task pre-training strategy, including diverse tasks such as span denoising, causal language modeling (CLM), and text-code comparison learning. Such a set of pre-training tasks helps the model learn richer representations from both code and text data, allowing for better migration and adaptation in various applications.

In terms of model architecture, CodeT5+ adopts a “shallow encoder and deep decoder” architecture. The encoder and decoder get initialized by pre-training checkpoints and connected to the cross-attention layer. By freezing the deep decoder and training only the shallow encoder and the cross-attention layer, the computational efficiency is improved while the performance of the model is maintained. In addition, CodeT5+ introduces mechanisms for adjusting instructions to better align with natural language instructions. This mechanism makes the model more flexible in understanding and following natural language instructions, thus better meeting user expectations when generating code.

The CodeT5+ model was trained using the expanded CodeSearchNet pre-training dataset, which contains nine programming languages, as shown in Table 1. The model was divided into two groups for pre-training, the first group being CodeT5P-220M, CodeT5P-770M, and the second group is CodeT5P-2B, CodeT5P-6B, and CodeT5P-16B. The first group is trained from scratch according to the architecture of T5; while in the second group, the decoders of the models are initialized from the CodeGen-mono-2B, CodeGen-mono-6B, CodeGen-mono- 16B models were initialized, and the encoder was initialized from the CodeGen-mono-350M model.

In terms of model pre-training, CodeT5+ adopts two stages for pre-training: in the first stage of pre training, the model undergoes pre-training for span denoising tasks and joint training for two CLM tasks, and uses a linear decay learning rate (LR) scheduler with a maximum learning rate of 2e-4. The batch size of the denoising task is set to 2048, while the batch size of the CLM task is 512. In the second stage of pre-training, the model adopted a strategy of equal weight contrastive learning, matching, and joint optimization of two CLM losses, and underwent 10 cycles of training. Set the batch size to 256 and the learning rate to 1e-4. The maximum length of the code and text sequence is set to 420 and 128, respectively. The model uses the AdamW optimizer weights decay to 0.1. At the same time, the mixed precision training technique of ZeRO Stage 2 and FP16 using DeepSpeed [10] is utilized to accelerate the training process.

When facing problems, Ego usually consults and organizes relevant information in the knowledge base to enhance the specificity and accuracy of the answers. In recent years, some researchers have attempted to incorporate knowledge bases into generative tasks and perform diverse fusion operations to improve the efficiency of algorithms. They proposed a hybrid neural dialogue model with both response retrieval and generation capabilities. Lewis et al. [11] proposed a RAG framework for knowledge intensive NLP tasks, which utilizes the DPR(Dense Passage Retrieval) algorithm to extract information from search results, concatenates the extracted information with the original input, and finally inputs the concatenated results into a generator for processing [12]. Experimental results have shown that this method can produce more specific and accurate results.

In order to fully utilize DPR technology and its advantages in natural language processing and information retrieval, this paper adopts DPR technology to achieve code extraction of Ego in the domain knowledge base. DPR uses a text encoder to encode the questions and answers in question and answer data separately to convert the input text into a dense vector representation. By calculating the similarity between the two vectors to evaluate their correlation, it achieves fast retrieval in large-scale text datasets.

This paper constructs an information enhancer specifically designed for machine learning code generation tasks based on DPR technology. In DPR, by using question answer pairs as training data, the model can learn how to accurately match the correlation between questions and answers, thereby improving the accuracy of retrieval. Two of the encoders used pre trained CodeBERT to obtain better vector representations. Its structure is shown in Fig. 6.

Figure 6.

The objective likelihood function of the enhancer can be expressed as equation (12): (12) L(qi,ci+,ci,l−,…,ci,n−l−)=−logesim(qi,ci+)esim(qi,ci+)+∑j=1n−1esim(qi,ci,j−) L\left({{q_i},c_i^ +,c_{i,l}^ -, \ldots,c_{i,n - l}^ -} \right) = - \log {{{e^{{\rm{sim(}}{q_i},c_i^ +)}}} \over {{e^{{\rm{sim(}}{q_i},c_i^ +)}} + \mathop \sum \nolimits_{j = 1}^{n - 1} {e^{{\rm{sim}}\left({{q_i},c_{i,j}^ -} \right)}}}}

Among them, q_i represents the i th natural language input, ci+ c_i^+ refers to the correct descriptive information fragment related to natural language q_i, ci,j− c_{i,j}^- represents the j th descriptive information block except for ci+ c_i^+ , n means the total number of samples, and sim stands for the calculation of dot product similarity. After being processed by an information enhancer, the processing method of Izacard et al [14] is referenced to splice and replace natural language inputs with descriptive chunks of information The process is shown in equation (13): (13) x′=x⊕y1⊕y2⊕…⊕yn x' = x \oplus {y_1} \oplus {y_2} \oplus \ldots \oplus {y_n}

Where x denotes the original input text, y_k denotes the k th spliced and replaced descriptive information block, ⊕ denotes the splicing and replacing operation, and x′ denotes the spliced and processed input text. The original input text for the CodeT5+ model is shown in Fig. 7 below.

Figure 7.

The above figure shows the original input text of the CodeT5+ model. Among them, task area represents the domain of the machine learning problem, dataset name represents dataset object's name, and algorithm name represents the algorithm object's name. In addition, the original input also includes module annotations related to machine learning programs, such as importing third-party libraries, loading and splitting datasets, model definitions, etc. The annotation texts of each module are connected with placeholders “[EKG]”.

When performing information augmentation, the relevant fields such as domain, dataset name, algorithm name, and placeholder “[EKG]” will be replaced by the algorithm selected by the Ego algorithm's decision-making ability and the relevant information retrieved by DPR, forming the input source data after replacement processing. Partial retrieval information examples and text replacement examples are shown in Fig. 8 and Fig. 9.

Figure 8.

Figure 9.

When DPR fails to retrieve the corresponding text, the CodeT5+ model will directly generate code and replace the corresponding part of the placeholder “[EKG]”. After DPR retrieval replacement and CodeT5+ model generation replacement, the original input will become a complete code sequence. The final example of code generation is shown in Fig. 10.

Figure 10.

The above methods combine the advantages of information retrieval and generative models. DPR can quickly and accurately retrieve relevant code fragments, providing rich contextual information and prior knowledge. The retrieved code snippets help CodeT5+better grasp the context and generate code that matches the task requirements.

After cleaning and preprocessing the crawled data, this article successfully screened 233 machine learning datasets and 1448 machine learning algorithms, which will be used for training models and analyzing user item interactions. As shown in Table 3.

a) Evaluation indicators. This article models the decision-making ability of Ego algorithm as a recommendation algorithm, and in recommendation algorithms, the recommended results are usually viewed as a classification problem, that is, whether users like the items recommended by the recommendation system. Therefore, this article adopts commonly used indicators, including AUC, Precision, Recall, F1 score, and NDCG.

b) Parameter settings.

For RippleNet, The object embedding and relation embedding dimensions are configured to 16, with a maximum of 3 hops, 10 epochs, and a batch size of 32, optimized using the Adam optimizer. The learning rate and regularization coefficients are determined via grid search, and the search spaces are {10-4, 5 × 10-4, 10-3, 5 × 10-3}和{10-5, 10-4, 10-3, 10-2};

For TCF, the dimension of the text embedding was set to 300, Multiply was used in GMF for linear computation, and 4 fully connected layers were used in MLP for nonlinear computation, and the outputs of GMF and MLP were connected by Concatenate of NeuMF.

c) Comparison experiment. We compare AD-EKG with KGNN-L[14] and KGCN [15] recommendation models

d) Ablation experiment. To investigate the validity of the algorithm, i.e., whether both graph structural information and textual descriptive information are helpful for recommendation, this paper sets up the following scenarios for Top-K evaluation:

Using only structural information (RippleNet);

The results of AD-EKG on the CTR prediction task and Top-K's recommendation are shown in Fig. 11 and Table 4, respectively.

Figure 11.

The experimental results from the experiments are presented in Table 4 and Fig. 11. From the metrics Precision, Recall, F1-score and NDCG, AD-EKG outperforms the model that does not use both information in the recommendation task. The Recall value of AD-EKG increases with the length of the recommendation list, which indicates that the model is better able to capture the user's interests and needs. The NDCG metric is a ranking quality and relevance to measure the performance of ranking models in recommendation algorithms, AD-EKG is also higher than traditional models in NDCG metrics, indicating that AD-EKG can provide more relevant and higher quality recommendation results.

In summary, the AD-EKG model outperforms the single method traditional model on the CTR prediction task and Top-K recommendation. This suggests that the simultaneous use of structural and descriptive message from the knowledge graph can significantly improve the effectiveness of recommendation models.

In order to verify the performance of the DPR technique on the code generation task, this paper constructs a dataset of questions related to machine learning program generation.

The dataset mainly consists of 122 question and answer data on machine learning image classification questions, as shown in Fig. 12 below. The dataset of the machine learning program constructed in this paper to generate relevant questions is shown in Fig. 12 above, where columns 3, 5, 14, 16, 17, and 18 of the file correspond to the dataset, algorithm, description of the algorithm, description of the dataset, type of the task, relevant questions, and the answers of the machine learning domain, respectively, and the detailed data about the question and answer section is shown in Table 6.

Figure 12.

In this paper, the CodeBLEU metric [16] and ROUGE [17] metrics are used for assessing the quality of the code generated by the model. The CodeBLEU metric is a variant of the BLEU (Bilingual Evaluation Understudy) metric [18], and the BLEU metric is calculated as follows: (14) BLEU=BP⋅exp(∑n=1NwnlogPn) {\rm{BLEU}} = {\rm{BP}} \cdot \exp (\mathop \sum \limits_{n = 1}^N {w_n}\log {P_n}) (15) BP={1ifc>re1−r/cifc≤r {\rm{BP}} = \left\{{\matrix{1 & {if} & {c > r} \cr {{e^{1 - r/c}}} & {if} & {c \le r} \cr}} \right.

w_n denotes the weight of the n -tuple and p_n is the precision of the co-occurring n -tuple. BP is a penalty factor used to ensure that the scoring takes into account the length of the generated sequence and does not just focus on how accurate the generation is.

CodeBLEU is based on BLEU, additional syntactic matching as well as semantic matching score items are introduced, and the final score is weighed by a certain proportion, and its calculation formula 16 is as follows: (16) CodeBLEU=α⋅BLEU+β⋅BLEUweight+γ⋅Matchast+δ⋅Matchdf {\rm{CodeBLEU}} = \alpha \cdot {\rm{BLEU}} + \beta \cdot {\rm {BLEU}_{weight}} + \gamma \cdot {\rm {Match}_{ast}} + \delta \cdot {\rm {Match}_{df}}

In BLEU calculation, different tokens have the same weight, and different tokens have different weights in CodeBLEU calculation. In equation (16), BLEU_weight is a weighted n-gram matching metric, similar to the BLEU computation; Match_ast is the similarity of the abstract syntax tree, which is used to measure the syntactic information of the code; and Match_df is the similarity of the semantic data flow, which takes into account the semantic similarity between the generated code and the reference code.

ROUGE metrics are mainly used to measure the degree of overlap between computer-generated code and reference code to evaluate assess the quality of automatically generated code. Commonly used evaluation metrics include ROUGE-N and ROUGE-L.

ROUGE-N mainly evaluates the code quality by calculating the number of n-grams that are the same in all the sentences in the automatically generated code and the reference code, and the proportion of them in the reference code. The detailed calculation formula 17 is given below: (17) ROUGE-N=∑S∈{Reference}∑gramN∈SCountmatch(gramN)∑S∈{Reference}∑gramN∈SCount(gramN) {\rm{ROUGE - N}} = {{\sum\limits_{S \in \{Reference\}} {\sum\limits_{{gram}_N \in S} {{Count}_{match}({gram}_N)}}} \over {\sum\limits_{S \in \{Reference\}} {\sum\limits_{{gram}_N \in S} {Count({gram}_N)}}}}

gram_N means that the length of the word is n, Count_match (gram_N) represents the frequency with which words of length n exist both within the automatically generated code and within the reference code, as opposed to Count (gram_N) which represents the frequency with which words of length n exist only within the reference code.

ROUGE-L counts the longest common substring that exists between the automatically generated code and the reference code to evaluate the overall coherence of the code, with Eqs. (18, 19, and 20) as follows: (18) Rlcs=LCS(X,Y)m {R_{lcs}} = {{LCS\left({X,Y} \right)} \over m} (19) Plcs=LCS(X,Y)n {P_{lcs}} = {{LCS\left({X,Y} \right)} \over n} (20) Flcs=(1+β2)LCS(X,Y)Rlcs+β2Plcs {F_{lcs}} = {{\left({1 + {\beta ^2}} \right)LCS\left({X,Y} \right)} \over {{R_{lcs}} + {\beta ^2}{P_{lcs}}}}

Equation (19) and equation (20) denote the calculation of recall R_lcs and accuracy P_lcs, respectively. The F_lcs in equation (21) denotes the final calculated ROUGE-L value. Where X denotes the text of the reference code, and its length is identified by m. Y denotes the text of the model-generated code, and its length is identified by n. LCS (X, Y) denotes the length of the longest common subsequence of X and Y.

The parameter β is generally set to a larger number, which is used to indicate that the calculation of P_lcs recall holds a larger weight in the calculation of F_lcs.

Comparison of the experimental results is shown in Table 7. It indicates that combining DPR technology with generative models is more effective in handling code generation problems than pure generative models when using the same parameter quantity model.

Analyze the results in Table 8. As the number of model parameters increases, CodeT5-EKG shows significant improvements in both CodeBLEU and ROUGE metrics. Compared to purely generative models such as CodeGen Mono and GPT Neo, CodeT5-EKG exhibits higher code generation accuracy and consistency at smaller parameter sizes. In conclusion, the comparative results in Table 8 show that combining DPR techniques with generative models has significant advantages in English code tasks.

The retrieval + generative model constructed in this article combines the efficiency of the retrieval model with the creativity of the generative model. This combination enables the model to better control the generated content and make the generated content more reasonable. Compared to pure generative models, retrieval + generative models require fewer parameters and computational resources, making them easier to train and deploy. However, this model also has some limitations. It relies on information from prior data for retrieval, so different prior knowledge needs to be stored for different fields or tasks. Secondly, this behavior of retrieval may lead to a lack of diversity in the generated content, which may not be as flexible as pure generative models in some cases.