Study of agricultural finance policy information extraction based on ELECTRA-BiLSTM-CRF

ELECTRA model [12] is an improved version of the dynamic pre-trained language model BERT, which is used to capture semantic information in agricultural finance policies. ELECTRA has greater learning representation, and since it employs two classifications of judges, it is not necessary to model the entire distribution of data, and it has greater operational efficiency and speed of convergence. Additionally, it can effectively improve the utilisation of information by using its own contextual predictions, which are appropriate for training a model with greater accuracy with fewer data samples. In addition to this, given the issue of possible Chinese character recognition errors during the OCR process, ELECTRA’s training for the generation of model replacement of masked word recognition is beneficial for improving the accuracy of the model in the case of incorrect recognition of words. When the model has error recognition words, entity annotation accuracy is improved.

Figure 3 illustrates the model structure of ELECTRA, which consists of a Generator and a Discriminator. The generator is used primarily to replace some words in a sentence, and the Discriminator is used primarily to determine whether each word in a sentence has been replaced, with the training process predicting all words, which is more efficient than BERT.

Fig. 3

The discriminator can easily determine whether the word has been replaced or not by using simple random substitution in ELECTRA. Therefore, ELECTRA trains the Generator using Masked Language Model (MLM) and then replaces the word with the Generator’s prediction. Eqs (1) and (2) show the prediction formulas for Generator Discriminator, respectively.

1 pG(xt∣x)=exp⁡(e(xt)ThG(x)t)exp⁡(e(xt)ThG(x)t)∑x′exp⁡(e(x′)ThG(x)t) $$\matrix{ {{p_G}({x_t}\mid x) = \exp \left( {e{{({x_t})}^T}{h_G}{{(x)}_t}} \right){{\exp \left( {e{{({x_t})}^T}{h_G}{{(x)}_t}} \right)} \over {\mathop \sum \limits_{{x^\prime }} \exp \left( {e{{({x^\prime })}^T}{h_G}{{(x)}_t}} \right)}}} \cr } $$

Where: h_G(x)_t is the post-encoding vector; t is the location e(x) is the embedding of the word x.

2 D(x,t)=sigmoid(wThD(x)t)

The loss function of ELECTRA includes the loss function LMLM of the generator and the loss function L_DiSC of the discriminator, which is formulated as shown in Eq. (3). The training loss function of the generator remains the MLM loss function because the generator replaces discrete words, which breaks the discriminator-to-generator gradient.

3 Loss=minθG,θD⁡∑x∈XLMLM(x,θG)+λLDiSC(x,θD) $$\matrix{ {Loss = \mathop {\min }\limits_{{\theta _G},{\theta _D}} \mathop \sum \limits_{x \in X} {L_{MLM}}(x,{\theta _G}) + \lambda {L_{DiSC}}(x,{\theta _D})} \cr } $$

In the training process of the ELECTRA model, the weight sharing and joint training with Discriminator models are adopted, which increases the learning and recognition difficulty of the Discriminator step by step. Part of Discriminator.

With ELECTRA’s pre-trained model, it is possible to extract semantic information from text by training it using the MLM method. The semantic recognition training, however, may not be sufficient for the resolution of the problem of near-form word errors when recognising the acquisition of information on the source-side agricultural finance policy. To improve the information extraction abilities of the Discriminator in specific scenarios, the image pre-training model has been further added to the text generation process of the Generator during model training. The specific steps are as follows.

Upon completing the mask on the word, the replacement word sampling is derived from Eq. (4).

4 x^i1=pG(xi∣xmasked)fori∈m $$\matrix{ {\hat x_i^1 = {p_G}\left( {{x_i}\mid {x^{masked}}} \right){\mkern 1mu} {\rm{for}}\>i \in m} \cr } $$

Where: m represents the set of masked words. On the other hand, the image samples corresponding to x_i represent vectorised using a neural network image pre-training model to obtain v(x_t), and Using the cosine similarity degree of the embedding vector, we can estimate the degree of similarity of the text in terms of glyphs.

5 sim(xi,xj)=v(xi)⋅v(xj)‖v(xi)‖‖v(xj)‖

Based on the cosine similarity degree of the text corresponding to the image embedding, the zigzag character of x_i is sampled from the entire font w and is denoted as 6 x^i2=pV(xi|w)fori∈m $$\matrix{ {\hat x_i^2 = {p_V}({x_i}|w){\mkern 1mu} {\rm{for}}\>i \in m} \cr } $$

In Generator, replacement samples are generated by replacing the samples of the generation model with zigzag characters with a certain probability to simulate the glyph error problem in OCR recognition.

7 x^i=δ⋅x^i1+(1−δ)⋅x^i2

Where: δ satisfies the 0-1 distribution. The generator can simulate semantic similarity as well as glyph similarity of the text through random substitution, enhancing the information mining capabilities of the Discriminator.

The Bi-LSTM combines a forward Long Short Term Memory (LSTM) with a backward LSTM. A LSTM Network is one of the most widely used recurrent neural networks, capable of learning the long-term dependent information associated with agricultural finance policy text sequences. In the process of training using agricultural finance policy text, gradient explosion and gradient disappearance problems may occur, but LSTMs are capable of resolving these issues. An LSTM consists of a forgetting door, an input gate, and an output gate.

As shown in Eq. (8), the forgetting door selectively forgets the input information of the previous cell node.

8 ft=σ⋅(Wf⋅[ht−1,Xt]+bf)

Where: f_t represents the output value of forgetting door, W_f represents the weight matrix, h_{t – 1} represents the hidden layer state at the previous moment, X_t represents the input at the current moment, and b_f represents the bias.

With the input gate, the input information of the current node is retained selectively, and the formula is shown in Eqs (9)–(11).

9 it=σ⋅(Wf⋅[ht−1,Xt]+bi) 10 C~t=tanh⁡(WC⋅[ht−1,Xt]+bC) $$\matrix{ {{{\tilde C}_t} = \tanh ({W_C} \cdot [{h_{t - 1}},{X_t}] + {b_C})} \cr } $$ 11 Ct=ft⋅Ct−1+it⋅C~t $$\matrix{ {{C_t} = {f_t} \cdot {C_{t - 1}} + {i_t} \cdot {{\tilde C}_t}} \cr } $$

Where: i_t represents the output value of the input gate, C~t ${\tilde C_t}$ represents the temporary state of the current cell node, and C_t represents the state of the current cell node.

As the output gate primarily outputs information from the current time node selectively, its equations are shown in Eqs (12) and (13).

12 ot=σ⋅(Wo⋅[ht−1,Xt]+bo) 13 ht=ot⋅tanh⁡(Ct)

Where: o_t represents the output value of the output gate and h_t represents the state of the hidden layer at the current moment.

LSTMs are capable of effectively filtering and storing information from memory units and of capturing dependency relationships over a longer distance. However, problems arise when LSTM is used to model the text of agricultural finance policy: it can only extract information from the previous text and cannot encode information from backward to front. In the process of extracting agricultural finance policy information, the information of anterior and posterior texts helps to identify relevant information. In this paper, a word vector output from ELECTRA is used as an input to BiLSTM, and the forward and backward LSTMs are then used to determine hidden information of the anterior and posterior texts of agricultural finance policy texts, respectively, and then the two information are stitched together and sent to CRF for the determination of agricultural finance policy information.

As discussed previously, the random field (CRF) is primarily responsible for capturing the dependency relationship between the preceding and following labels, which correspond to the text of agricultural finance policy, and leveraging the global knowledge of label sequences to better predict the labels of agricultural finance policy entities. The CRF is a discriminative, probabilistic, undirected graph model that is commonly used to analyse and label sequential data. The maximum entropy model and the hidden Markov model have been combined in this model to achieve better results in sequence labelling tasks, such as word separation, lexical labelling and named entity recognition.

In this paper, we apply the linear chain CRF, which is the conditional probability distribution P(Y ∣ X) model that fully satisfies the Markov property when both the observation sequence X and the state sequence Y are linear chains. Eqs (14) and (15) show the parameters for the linear chain CRF.

14 P(Y∣X)=1Z(X)⋅exp⁡∑k=1Kωkfk(Y,X) 15 Z(X)=∑Yexp⁡∑k=1Kωkfk(Y,X)

Where, the observation sequence X corresponds to the sequence of agricultural finance policy text in this paper; the output sequence Y corresponds to the entity label category of the agricultural finance policy text sequence correspondence; Z(X) represents the normalisation factor; f(·) represents the characteristic function; and ω represents the weight corresponding to the characteristic function. The conditional probability model is obtained through maximum likelihood estimation using a training set of agricultural finance policy texts P(Y ∣ X). During the prediction of agricultural finance policy text, the dynamic programming algorithm is used to find the optimal output sequence that maximises the conditional probability P(Y ∣ X) for a given sequence of agricultural finance policy text, to determine the entity labels that correspond to the agricultural finance policy text. An entity label is obtained for the text corresponding to the entity.

It is necessary to train and predict an information extraction model based on an annotated database of agricultural finance policies to extract information on agricultural finance policies. By utilising existing structured information on agricultural finance policies on the Internet, we can reduce the workload of manual annotation during the training set construction process. Based on the structured agricultural finance policy information data, we apply the BIO annotation method to automatically annotate the field information of each important aspect of the agricultural finance policy. Meanwhile, the collected texts of agricultural finance policy are categorised and stored separately. The model is first fine-tuned using large batches of automatic production data, and then fine-tuned twice using small batches of real agricultural finance policy data.

This experiment uses three evaluation metrics: precision rate (precision), recall rate (recall) and F1 value (F1-score). The F1 value is a combined evaluation metric for precision rate and recall rate. Eqs (16) and (17) provide the formulas for precision and recall rates.

16 precision=TPTP+FP 17 recall=TPTP+FN

Where, TP is the number of information entities that were correctly identified by the model, is the number of irrelevant information entities that were identified by the model, and FN is the number of relevant information entities that were not identified by the model.

Eq. (18) shows the formula for the F1 value (F1-score).

18 f1=2⋅precisionprecision+recall

The experimental platform utilises an NVIDIA Tesla T4 graphics card with 32 GB of video memory and an Ubuntu 18.04 operating system. In the Electra model, the BERT model is used as the backbone, with 12 layers, 768 hidden units and 12 multi-headed attentions; the BiLSTM input dimension is 768 while the layer dimension is 128. By using the ELECTRA-based Chinese model initialisation weights, the model was fine-tuned in two stages based on both an automatically generated dataset and an OCR real dataset. Adam is the optimiser, the initial learning rate is 0.001 and the number ratio of the training set to the test set is 7:3.

To compare and evaluate the performance of the information extraction model of agricultural finance policy, we developed three models: BERT-BiLSTM-CRF, BiLSTM-CRF, and BiLSTM. One of these is the BiLSTM model which uses Word2Vec as input for pre-trained word vector recognition. With input Word2Vec, the BiLSTM-CRF model adds a CRF layer to BiLSTM. The BERT-BiLSTM-CRF takes advantage of the same neural network structure as that of the proposed model, as well as the BERT-base Chinese pre-training model for initialisation.

Figure 4 illustrates the training of the four models in 100 rounds of epochs. As shown in the figure, the F1 scores of the two models that use the BERT model for word embedding are significantly higher than those of the two models that use Word2Vec for word embedding. Due to the same neural network structure used by both ELECTRA and BERT, the convergence speed of the two models using word embedding is essentially the same, and ELECTRA showed a slightly better accuracy than BERT.

Fig. 4

Figure 5 illustrates the performance of the four agricultural finance policy information extraction models on the test set of the agricultural finance policy F1-score. Compared to BiLSTM, BiLSTM-CRF and BERT-BiLSTM-CRF, the BERT-BiLISTM-CRF has the highest F1 score on the test set for the agricultural finance policy. Contrary to the BERT-BiLISTM-CRF, which is based on the BERT model, the model proposed in this paper shows better performance, indicating that the pre-training task developed for the problem of agricultural finance policy labelling is more appropriate.

Fig. 5

Figure 6 illustrates the evaluation results of the four models when applied to the same experimental environment and training parameters. According to Figure 6, both models using ELECTRA for word embedding outperform those using BERT or those without BERT. The accuracy rate of the agricultural finance policy information extraction model after 100 rounds of training is 92.02%, the recall rate is 88.31%, and the F1 value is 89.73%, and all the indicators have reached high levels. The results of the study show that when building the agricultural finance policy information extraction model, the model has a stronger ability to extract features as well as higher information extraction accuracy.

Fig. 6