Error Analysis and Instructional Strategy Adjustment in a Corpus of English Language Learners

Recurrent Neural Networks (RNN) are a class of recurrent neural networks that take sequence data as input, recursively in the evolutionary direction of the sequence, and all nodes are connected in a chain fashion [20]. Due to the recursive nature, recurrent neural networks can represent sequences of arbitrary length as vectors of fixed length while focusing on the structured properties of the input.

RNN networks mainly involve some formulas as follows: 1st=f(U⋅xt+W⋅st−1) 2ot=σ(V⋅st)

Equation (1) contains the parameter matrices U and W, the time markers t, the input vector x, and the intermediate hidden state s. The hidden state s_i of the current time step is derived from the input x_i by co-computing with the state s_t–1 of the previous time step, while the output vector o_i in Equation (2) is derived from the current s_i by linear transformation and activation. The parameters of each time step are shared, and the output vector of the last time step can contain the information of the whole sequence.

In the training process of recurrent neural networks, the classical BP backpropagation algorithm is usually used to update the parameters forward from the last moment t. During the BP derivation process, too long sequences can have vanishing or exploding gradients. In order to solve the problem of long distance dependence of recurrent neural networks, a RNN variant modeled Long Short-Term Memory Network (LSTM) with gating structure has been proposed.

The biggest difference with the RNN hidden layer unit is that there is an extra memory unit C, and its main gating structure consists of a forgetting gate f, an input gate i, and an output gate o. The following briefly describes the computation process of the hidden layer in LSTM.

In the formula (3) of the forgetting gate, the output h_t–1 of the previous time-step unit and the input x_i of the current unit are linearly transformed, and the sigmoid activation function controls their values in [0, 1] to control the degree of forgetting the information of the previous unit: 3ft=σ(Wf⋅[ht−1,xt]+bf)

The input gates are computed as follows, the tanh function in Eq. (4) produces a new candidate vector C˜i , and the input gating i_i in Eq. (5) produces a value within [0, 1] for each item in C˜i to control how much new information is added. Finally C_i in Eq. (6) is the integration of the previous time step C_i = 1 and the new input C˜i , which completes the update of the previous memory cell: 4C˜t=tanh(WC⋅[ ht−1,xt ]+bC) 5it=σ(Wi⋅[ht−1,xt]+bi) 6Ct=ft⋅Ct−1+it⋅C˜t

The final output gate o_i is further filtered for the current cell state C, computed as shown in Eqs. (7) and (8) below, where the output gate produces a value of [0, 1] for each item of the cell state, controlling the extent to which the cell state is filtered: 7ot=σ(Wo⋅[ht−1,xt]+bo) 8ht=ot⋅tanh(Ct)

LSTM solves the RNN long-range dependency problem well with three gating structures and achieves better results. However, the complexity of the model is also significantly increased, the model has more parameters, more training data is needed to make the model fit, and the training time of the model is also increased a lot more than RNN.

The GRU unit contains the following main calculation steps: 9rt=σ(Wr⋅[ht−1,xt]) 10h˜t=tanh(Wh⋅[rt⋅ht−1,xt]) 11zt=σ(Wz⋅[ht−1,xt]) 12ht=(1−zt)⋅ht−1+zt⋅h˜t

Firstly, Equation (9) indicates that the hidden state h_i–1 of the previous time step and the input x_i of the current time step are linearly transformed, and after the sigmoid activation function processing, a reset gate r_t with a value in the range of [0, 1] is obtained. The role of the reset gate r_t is equivalent to a proportionality factor, which determines the amount of information in the previous hidden state that can be retained, and the retained information h˜ , which is obtained by the calculation of Equation (10). The updating gate z_i in Equation (11) is calculated in the same way as the reset gate r_i, and z_τ can simultaneously control the amount of information in the previous time step h_τ–1 and in the h˜τ′ obtained through the reset gate, as seen in Equation (12), the closer z_i is to 1, the more the information of the previous time step state h_i–1 is forgotten. The state h_r of the current time step is finally obtained.

Sequence to Sequence (Seq2Seq) framework is commonly used for modeling text generation tasks and has achieved great success in areas such as Machine Translation, Speech Recognition, Text Summarization, Question and Answer systems, etc. [21]. The Seq2Seq framework usually consists of two constructs, an Encoder and a Decoder, whose inputs and outputs are both a sequence. In Encoder, the input sequence is usually converted into a fixed-length intermediate vector, and then the intermediate vector is converted into a result sequence by Decoder. A brief structure diagram of the Seq2Seq framework is shown below in Figure 1. The most important feature of this framework is that the output sequence is of variable length, e.g., the input sequence {x₁, x₂, …, x_m} is of length m and the output sequence {y₁, y₂, …, y_n} is of length n in the figure, so it has great flexibility. Decoding usually starts from the symbol indicating the beginning such as <bos> until the symbol indicating the end <eos> is decoded (where bos denotes the beginning of sentence and eos denotes the end of sentence).

Figure 1.

Seq2Seq framework structure

Grammatical error correction task can also be regarded as a generative task, the input of the encoder side is the sentence containing grammatical errors, the output of the decoder side is the sentence predicted to be generated by the model after correction, at the same time in the correction of the error may be inserted or deleted some words resulting in the original sentence and the target sentence length inconsistency. For example, the sentence "I always study English in morning." The article "the" is missing from "in morning", and the Seq2Seq framework can be added by decoding the word "in" at the next decoded position when decoding to the word "in". This method of predicting the generation of indefinite-length sentences can achieve the correction of grammatical errors contained in the original sentence in the process of generating sentences.

There are fewer publicly available manually annotated corpora for grammar error correction tasks in China, and this subsection focuses on CLEC, which is used in this paper.

The CLEC corpus collects English writing compositions from students with different English proficiency levels, including secondary school students, university English level 4 and 6, and professional English level 4 and 6, corresponding to the five categories from ST2 to ST6, and containing more than one million words in total. The purpose of this corpus is to observe and analyze Chinese students’ verbal errors in the use of English through quantitative and qualitative methods, so as to provide useful information for the development of English education in China. For the grammar correction task, the grammatical errors in the CLEC corpus reflect the English grammar usage habits of Chinese students, and the use of the CLEC corpus to train and test the grammar correction model allows the grammar correction model to be better applied to the grammar correction scenarios of Chinese English learners.

CoNLL2013 and CoNLL2014 are two competitions held specifically for the problem of English grammatical error correction, which gave a clearer definition of the problem of English grammatical error correction and attracted many teams in the field of grammatical error correction to participate. It also promoted the enthusiasm of academics in researching this problem, which led to a certain development of grammar error correction.The task given in CoNLL2013 was to correct five common grammatical errors in English, including prepositional errors, noun singular-plural errors, qualifier errors, subject-verb agreement errors, and verb morphology errors. In CoNLL2014, on the other hand, the 28 most common grammatical errors in English are corrected, which is more difficult than CoNLL2013 but also more practical to the problem of English grammar correction. In this paper, the problems in CoNLL2014 will be studied, and natural language processing techniques and methods will be used to improve the effect of grammatical error correction.

The open source tool MaxMatchscorer implements the MaxMatch algorithm, and gives a simple and easy to use, you can directly use the tool to evaluate the effect of model error correction. The following is an introduction to the principle of MaxMatch algorithm.

Correction rate P, the degree of correctness of the model to correct the sentence, is defined as: 13P=∑i=1n|gi∩ei|Σi=1n|ei|

Correction rate R, the proportion of errors corrected by the model for all errors that should be corrected, is defined as: 14R=∑i=1n|gi∩ei|Σi=1n|gi|

The numerator of both equations above is the number of sentences in which the model correction result matches the reference answer, the denominator of the correction rate P is the number of sentences modified by the model, and the denominator of the correction rate R is the number of all sentences in the reference answer that need to be corrected.The key evaluation metric in MaxMatch is F_0.5, and the combined P and R are considered, and the equations are defined as follows: 15F0.5=(1+0.52)*R*PR+0.52*P

From the above equation, it can be seen that the algorithm amplifies the percentage contribution of the correction rate and reduces the percentage contribution of the correction rate. It makes the overall presentation that whenever a sentence is corrected it is better to be correct, otherwise it is preferred not to make corrections, penalizing the phenomenon of correcting an otherwise correct sentence.

Aiming at the fact that the English grammar error correction problem has certain similarities with machine translation, and the seq2Seq model has made many breakthroughs in the field of NLP, in this paper, we will choose seq2Seq as the basic model for grammar error correction problem solving, and improve the training efficiency and model error correction ability of the basic model of seq2Seq, so as to improve the effect of English grammar error correction. The main improvement points are described in detail below.

In the seq2seq model, the encoder forms the input sequence into a fixed-length context vector, and a large amount of detailed information is lost during the computation of the context vector, which becomes more obvious when the sentence length exceeds a certain limit. Therefore, scholars have carried out a series of studies to address this problem, which was eventually inspired by research work on human vision.

Figure 2 shows the attention model, according to the calculation range division of the input sequence, the attention mechanism can be divided into two kinds of local and global, in this paper, we will use Soft Attention in the global computation combined with seq2seq to deal with the problem of grammatical error correction, so the following will be introduced to Soft Attention [22].

Figure 2.

Soft Attention model

The value of weight α_ij is determined by the i – 1 nd hidden state s_i–1 and each of the hidden state variables in the input, calculated as follows: 16αij=exp(eij)Σk=1Txexp(eik) 17eij=α(si−1,hj)

Deep neural networks are powerful, but their requirements for machine computational power are high, and the improvement of training efficiency plays a key role in model optimization and improvement [23]. Theory shows that normalizing the input activation parameters of neurons can reduce the model training time. The (BN) introduced in CNN by normalizing the input activation parameters of any neuron in each layer of the network, the activation parameters conform to the standard normal distribution after the transformation, which makes the activation parameters fall in the region of the slope of the nonlinear function, the gradient is larger, and the model convergence speed is faster.

However, since the BN operation depends on the mini-batch first-order and second-order statistics, which are related to the size of the mini-batch, and the RNN input length is variable, the BN cannot be directly used in the RNN. In order to improve the RNN training efficiency, researchers proposed (LN), which can significantly reduce the model training time and various RNN networks after layer normalization outperform the original network.

LN is normalized for the training samples, which is performed independently and in parallel with other sample data, so it is not as demanding for the data distribution as BN, and the training difficulty decreases drastically.LN is a kind of transversal normalization, which maps the inputs of neurons of an entire layer in an RNN into the same distribution by summing up the inputs of neurons in the layer with the computation of the variance, and all the hidden elements share the same normalization terms μ and σ: 18μ=1H∑i=1Hxi 19σ=1H∑i=1H(xi−μ)2

Since computers can not understand natural language, so the use of computers for text processing must be vectorized into numbers that computers can understand. The simplest way to encode is one-hot matrix representation, that is, with a fixed length of the size of the vector represents a word, the vector length for the size of the word list, only in the word appears in the position of 1, and the other are 0, such as “book” using [0, 0, 1, 0, 0, 0, …] that the method is simple, but the disadvantages are obvious. The method is simple, but the disadvantages are also obvious, the data dimension is too large, too sparse, and the relationship between words is difficult to reflect.

For the modification suggestions submitted by users, taking into account the varying levels of English proficiency of users, it is necessary to screen these utterance suggestions to identify which are correctly modified utterances and which are incorrect utterances. This is related to the process of re-learning and re-training of the system, and belongs to the key module of the system. This filtering process is actually comparing the modified statements given by the system and the modified suggested statements given by the user which one is more credible, then the sentences need to be scored, and those with higher scores can be considered more probable to be the sentences with no grammatical errors, so when the user suggests to modify the sentences with a score lower than the system’s score of the modified sentences will not be adopted. Therefore, in order to effectively filter invalid suggested text, the n-gram grammar model is used to score the sentences. The n-gram model is described below.

Using a string S to represent a sentence, the n-gram model models the string S. P(S) denotes the probability of occurrence of the sentence. Words are the basic units in the n-gram, and a sentence consisting of m words has probability P(S) = P(ω₁, ω₂, ⋯, ω_m). According to the chain rule, the probability of a string S is calculated as follows: 20P(ω1,ω2,⋯,ωm)=P(ω1)P(ω2|ω1)P(ω3|ω1,ω2)⋯P(ωm|ω1,⋯ωm−1)

According to the above equation, it can be seen that the i st word ω_i probability is related to the word ω₁ω₂ ⋯ ω_i–1 before that word, and there is a dependency relationship between words, and the i th word depends on the previous i – 1 words. But obviously Eq. (20) is very difficult to compute, and the computation is particularly large. Moreover, due to the dataset, many combinations do not appear when we train the model, and the model parameters are difficult to obtain.

For this reason, the Markov assumption is introduced that the current word is only related to the previous few words, which can simplify the calculation process as: 21P(ωi|ω1,⋯,ωi−1)=P(ωi|ωi−n−1,⋯,ωi−1)

For the n-gram language model, it is common that n can be taken as 1, 2, or 3. A start and end flag is usually added before and at the end of the sentence to facilitate uniform computation.

When n=2, bigram is: 22P(ω1,ω2,⋯,ωm)=∏i=1mP(ωi|ωi−1)

When n = 3, trigram is: 23P(ω1,ω2,⋯,ωm)=∏i=1mP(ωi|ωi−2ωi−1)

It is sufficient to make Eq. (23) obtain the maximum value according to the maximum likelihood estimation method. These are the basic elements of the N-metric grammar. In the training corpus, the value of P(w_i | w_i–2w_i–1) can be estimated with the following formula: 24P(wi|wi−2wi−1)=c(wi−2wi−1wi)c(wi−2wi−1)

After training the model it is possible to calculate the probability and perplexity of the sentence, and the concepts of probability and perplexity are first briefly explained below.

Probability, as the name suggests, is the frequency of occurrence of the sentence, according to the basics of the N-meta grammatical model introduced earlier: P(S) = P(ω₁, ω₂, ⋯, ω_m).

Perplexity, the basic idea is to give normal sentences to the larger the probability value of the language model is better, after training the model, use the model to score the test set, at this time, the test set are all normal sentences, then the larger the score of the model is better. The relevant formula is as follows: 25PP(S)=P(ω1,ω2,⋯,ωm)−1m=1P(ω1,ω2,⋯,ωm)m

It can be rewritten according to the chain rule: 26PP(S)=∏i=1mP(ωi|ωi−n−1,⋯,ωi−1)m

From the formula, it can be seen that the smaller the confusion degree is, the larger the sentence probability is. Comparing probability and perplexity, it can be found that the calculation of probability is greatly influenced by the length of the sentence and the number of words, while perplexity can effectively reduce this influence, so perplexity is often used instead of probability to judge the sentence and model. Therefore, in this paper, the confusion degree is also used, but for the sake of convenience and semantic comprehensibility, probability is sometimes used instead.

In order to demonstrate the effectiveness of the attention mechanism proposed in this paper, this subsection compares the error correction effects of the model without the attention mechanism with the present model with the attention mechanism, as shown in Fig. 3. The values in the figure are the P, R, F0.5 results obtained on the CoNLL-2014 test set. It can be seen that the model with the addition of the corresponding attention mechanism is significantly better than the model without attention on all five types. Among them, the attention mechanism is the most helpful for the noun singular-plural model, which is 25 more effective than the one without the attention mechanism on the F0.5 metric.

Figure 3.

The attention mechanism is relatively effective

This subsection compares the error correction results of the three classification models, this paper’s model, the CUUI model, and the deep contextual model, on five error types, namely, articles, prepositions, verb morphology, noun singular-plural, and subject-verb agreement, as shown in Figure 4. It can be seen that this paper’s model achieves the highest results in the other 4 types except for the preposition type, and in the F0.5 metric, the difference between the classification results of this paper’s model for coronal, verb morphology, noun singular and plural, and subject-predicate agreement is 6.6, 18.1, 8.0, and 8.0, when compared to the deep contextual model. And the progress is relatively obvious. In terms of preposition types, the model in this paper is slightly inferior to the deep contextual model, probably due to the large number of preposition classes, the deep model still can’t get a better grasp of the collocation laws.

Figure 4.

Comparison of results based on classification

Two classes were selected as experimental subjects in college J. The experimental class used the corpus error analysis constructed in this paper for English language learning, while the control panel used the traditional English language learning method for 15 weeks.

A pre-study test was administered to these two classes when they first entered their junior year of college, and the results showed that students in both classes had more grammatical errors in English writing, concentrating on prepositions, verbs, and conjunctions, and both had many common problems. After entering the third year of college, students were more concerned about improving their English performance, and after two years of study, students’ reading and completing the blanks were more correct, which was greatly related to the fact that English teachers in the school emphasized reading. At the stage of freshman and sophomore years, students practiced reading and gap-filling exercises a lot, but less practice in written expression and even less correction of errors, and usually memorized model essays and wrote essays for exams, so there are more grammatical errors in students’ writing, and the present study carries out an action research on the treatment of grammatical errors in high school students’ English writing according to the problem of grammatical errors in students’ writing.

Before, after the first, second and third rounds of this action research, tests will be given to the students of both classes, and then the test scores of both classes will be analyzed according to the data analysis method, so as to judge the effectiveness of the grammar correction material bank constructed in this paper on the students’ written expressions in English.

By correcting the pre-test paper, the results of the pre-test paper according to the eight lexical aspects examined in grammar [prepositional errors, pronoun errors, coronal errors, noun errors, adjective errors, adverb errors, verb errors and conjunction errors (syntactic errors)] are shown in Table 1.

In general, it seems that the students’ error rate in pronouns is low, but there is a compound structure, the students will be confused by other grammatical points, ignoring the use of pronouns in the wrong way, in topic 13, 78.15% of the students revised correctly, but in the other topic (topic 11), the correct rate is only 34.79%, and most of them directly ignored not to revise the error here and did not understand the grammatical examination of this sentence.

Table 2 shows the statistical scale of the sample mean of the writing pre-test, and the score rate of the essay fill-in-the-blank is higher, which is 12.9352. During the period of freshman and sophomore years, the students practiced a lot of grammar fill-in-the-blank by using the English grammar correction model in the corpus, and they mastered this type of question well, and the prompt words of the essay fill-in-the-blank questions are more obvious, which makes it easier for the students to write the correct answers compared with short-text corrections.

Table 3 shows the results of the pre- and post-tests of the first round of action research. Before and after the first round of action research, the score rate of the five types of questions, such as prepositional, pronouns, nouns, predicate verbs, and parallel conjunctions, increased, but the score rate of the five types of questions, such as coronal, adjectives, adverbs, nonpredicate verbs, and subordinate conjunctions, on the contrary, declined. The type with the highest increase in score rate was the noun category, which reached 29.89%, and the type with the highest decrease was the adjective category questions, which reached -31.46%. Taken together, before and after the first round of action research, although the students’ individual category score fluctuation values varied greatly, the final average score had little ups and downs and was more influenced by randomness.

A paired sample t-test was conducted on the grades before and after the first round of action research. The value of significance was 0.945, indicating that there was no significant correlation between the changes in pre- and post-test scores and the first round of action research. Combined with the fluctuations in the score rates of the various types of grammar questions, it can be concluded that the first round of action research using the corpus for error analysis in EFL did not have a significant impact on the students’ knowledge of grammar.

Table 4 shows the results of the pre-test and post-test of the second round of action research. Before and after conducting the second round of action research, the score rate of the grammar topics in all categories increased, except for the adverbial category, where the score rate decreased slightly. Among them, the type with the highest increase in the score rate is the predicate verb category, which reaches 32.67%. Taken together, the students’ grades in the grammar questions before and after the second round of action research on English language learning using the error analysis in the corpus constructed in this paper have increased significantly. The fluctuation value of the average score also rose more significantly compared to the first round of action research.

A paired samples t-test was conducted on the scores before and after the second round of action research. This time, the t-value was 3.039 and the significance value was 0.0154, indicating that the changes in the pre- and post-test scores were significant and statistically significant. Combined with the fluctuations in the score rates of the various types of grammar questions, it can be concluded that the second round of the action research significantly improved the students’ mastery of the various types of grammar.

Table 5 shows the statistics comparing the pre and post-test scoring rates for the third round of actions, with significant increases in the pre and post-test scoring rates for Topics 1 and 4 showing a significant improvement in the students’ understanding of the use of nouns. Topics 2, 3 and 7 showed an increase in score rate of 9.11%, a decline of -8.40% and an increase of 25.50%, respectively, which indicates that students’ understanding of coronal use is not always consistent and may need to be strengthened in some areas. Adverbs were examined in Topic 5 and the score increased from 86.93% to 100%, showing that students have a basic grasp of the use of adverbs. Adjectives were examined in Topic 6, and there was a 21.03% increase in the score, showing some improvement in the mastery of adjectives. Prepositions were examined in Topics 8 and 9 with an increase in score of 13.31% and 30.42% respectively, showing a significant improvement in students’ understanding of the use of prepositions. Verbs appeared in Topics 10, 12, 14, 15 and 19 with a range of improvement in scores from 17.1% to 25.36%, which shows some improvement in students’ understanding of verb usage. Pronouns were examined in Topics 11 and 13 and there was an increase in score of 34.49% and 17.28%, which shows a greater improvement in students’ use of pronouns. Conjunctions were examined in Topics 16, 17, 18, and 20, and there was an increase of more than 10% in the score rate, which shows a greater improvement in students’ understanding of the use of conjunctions.

Taken together, the majority of students improved their understanding of all types of lexemes. Although there was a decrease in scores on one quiz for articles, there was also an increase in scores on other quizzes, and it is possible that this is an isolated phenomenon. These data show improvement in students’ grammatical understanding and application.

Table 6 shows the paired samples t-test for the comparison analysis of the pre-test and the writing post-test, which showed that the post-test, compared to the pre-test, improved by an average of 5.0645 points for single-sentence corrections, 3.0354 points for short-text corrections, 3.3214 points for one-sentence fills in the blanks, 1.7258 points for one-sentence translations, 3.3454 points for essay fills in the blanks and 1.4598 points. Although the increase in the scores of each question type is different, it proves that students have made some progress.

In summary, the post-test performed better than the pre-test in all of the items mentioned. The t-value proves that such an improvement is statistically significant and the significance value of 0 for each item also indicates the significance of the improvement. This means that all the instructional activities, be it single sentence correction, short correction, single sentence fill in the blank, single sentence translation, essay fill in the blank, or written expression have achieved effective improvement.

Due to the differences between English and Chinese languages, students often make mistakes when using English because of the influence of mother tongue transfer, therefore, it is quite important to teach English grammar with appropriate use of comparative method so that students can understand the differences between English and Chinese languages.

In the process of grammar teaching, if only through the words, phrases, sentences and other static forms of teaching, repeated drills to make students memorize the grammar rules, this will only make students learn boring, and can not really understand the rules of grammar, can not flexibly use the language. Combined with the context of teaching grammar, that is, in teaching according to the specific teaching content and characteristics of the design of different contexts, the abstract rules of grammar into concrete language facts, in the corresponding context for students to actively feel and discover the rules of grammar, so that students will really learn and remember the rules of grammar, to avoid inappropriate generalization of the rules of grammar, to reduce the grammatical errors within the language.

The task-based teaching method emerged in the century. Task-based language teaching advocates letting students do things, i.e., mastering language through accomplishing various tasks, and adopting the method of letting students discover, generalize and summarize to cultivate their ability of independent learning and inquiry. The core idea of task-based teaching is to simulate the kinds of activities that people engage in when they use language in school life and society, to combine language teaching with language use, and to let learners actively participate in the attempts to communicate in the target language, so as to cultivate students’ comprehensive language competence. Advocates of task-based language teaching believe that grammar teaching should not be rejected, but should be emphasized instead. They believe that classroom instruction should teach the forms of grammar and how these forms can be used for communicative purposes.