A sentiment analysis method based on bidirectional long short-term memory networks

The model used in this experiment is the pre-trained GloVe model, a word vector trained based on 6 million elements of data, which is a word vector expression with functions similar to late semantic analysis (LSA) and Word2vec [13]. GloVe model characterises word vectors as real numeric vectors to vectorise words, so as to contain as much and accurate semantic and grammatical information between vectors as possible. The core of GloVe model is to construct word embedding matrix and process word vectors for each word in the obtained text. The dimension of this model is 300 dimensions, and the semantic similarity of text is often expressed by the similarity of vector space. Each word corresponds to a word vector in this model. If the words in the text are semantically similar, then the distance in the word vector is similar.

Glove, LSA and Word2vec are common methods to obtain embedded matrices. LSA is an early word vector representation tool. The dimension of large matrix is reduced by LSA based on singular value decomposition (SVD). However, due to the high complexity of SVD, its calculation process consumes time and is not friendly to computers with poor performance. In recent years, the Word2vec model is also used more in the process of deep learning, but since it contains 3 million word vectors, the word vector matrix is too large. Moreover, Word2vec does not use global co-occurrence, that is, it does not make full use of all corpora. Word2vec contains two structures, CBOW and skip-gram. CBOW model lacks the relationship between words in the whole sentence due to the direct addition of word vectors in the context, resulting in the lack of word relationship information. The skip-gram model is directly trained. Since this algorithm uses intermediate words to predict adjacent words, it is easy to get too much weight for high-frequency words. More importantly, the two models update the word vector with the information in one window at each training, but Glove is based on the global corpus (collinear matrix), that is, multiple windows, and therefore, the speed of model training is accelerated. Therefore, a more manageable matrix glove model completed by pre-training is used for training in this paper.

As a kind of unsupervised learning, deep learning does not need manual annotation in front of a large number of data elements. Its emotional classification of data is through its learning and model training. Therefore, it is only necessary to train the word vector after data pre-processing.

Data set pre-processing consists of the following sequence of tasks: input text, delete special symbols (punctuation marks, brackets) other than English words in the text and convert uppercase letters to lowercase. Since the computer cannot directly understand the text content, it is necessary to quantify the text into a numerical form so that the computer can understand through machine learning. Each word in the text can be represented by a vector, and then a word vector matrix is obtained through Glove model.

The word embedding word vector adopted in this experiment is proposed by Pennington. In order to solve the problem of sparse vector, this method adopts distributed mapping of words from high latitude space to low dimensional space. There are two commonly used models: CBOW and Skip-gram. Intuitively, Skip-gram predicts the context given the input word. CBOW is used to ascertain the given context to predict an input word, as indicated in Figure 1.

Fig. 1

In the process of deep learning, due to the need to train a large amount of text data, an overfitting phenomenon is often observed. Generally speaking, for example, after training the characteristics of a Persian cat with a large amount of data, the result is a cat. When taking the civet cat as a test, there is too much difference between the civet cat and the Persian cat due to the existence of overfitting, and thus the result says that the civet cat is not a cat. The over-fitted model has no practical value. In the experiment, in order to solve this problem, the integration method is usually used, that is, multiple models are combined to improve the accuracy. However, it is very difficult to train an algorithm to follow this method; it is also very time-consuming to test multiple models, and, additionally, doing so will result in excessive resource-consumption.

The core of the improved idea of neural network, proposed by Pennington [13] in his early years, is that in the training process of neural network model, a neural network unit is irregularly removed or lost from the neural network. It is worth noting that this discarding is only temporary, and thus some neurons in the model can be independent of other neurons, which greatly reduces and weakens the synergistic fitting and generalisation between neuronal features.

Spatial dropout is a dropout method proposed by Tompson et al. Through experiments, it can be found that the dropout method is irregular and will randomly set some elements of the matrix to zero, while the spatial dropout will randomly set some regions to zero. The spatial dropout method in the Keras module is also very effective in solving the overfitting problem.

LSTM is a variant of RNN, and it is a higher-level RNN. With the strong context connection ability of a memory unit in a long text, RNN refers to a sequence whose current output is related to the output before this time. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer in a state of non-connection, but are connected, and the hidden layer input includes not only the input layer output but also the hidden layer output at the previous time. The RNN model is shown in Figure 2.

Fig. 2

As can be seen from Figure 1, the value of s depends on two aspects: (1) input vector x and (2) the value of the hidden layer at the previous time. Eqs. (1) and (2) represent the calculation process of the recurrent neural network: (1) Ot=g​(v⋅St) {O_t} = g\left( {v \cdot {S_t}} \right) (2) St=f​(u⋅Xt+w⋅St−1) {S_t} = f\left( {u \cdot {X_t} + w \cdot {S_{t - 1}}} \right) Where: O_t represents the final output result, v and u are the weight matrixes respectively, representing the weight matrix from hidden layer to output layer and from input layer to hidden layer. S_t represents the value of the hidden layer, X_t represents the input value at the current time, and w is the weight matrix, representing the last S_t as the weight input this time.

RNN has some defects in the semantic understanding of long text, and thus this paper uses the LSTM model. The LSTM network adds a long-term state based on the RNN model. The hidden state of neurons is also called cell state, which is often represented by the symbol ‘C’. The core of LSTM is these gate structures. The concept of the gate in cell state is introduced below.

There are three gates in LSTM. These gates can effectively deal with some problems of gradient explosion and gradient disappearance caused by the RNN. Forget gate is used to delete excessive information in the cell at the previous time; the input gate is used to determine which information should be added to the cell as input at this time; the output gate is used to determine the output of information stored in the cell at the current time. Figures 3 and 4 describe the LSTM model and gate structure.

Fig. 3

Fig. 4

The calculation formula for each LSTM cell is as follows:

Forget gate: for each new input, the LSTM will decide which memories to be forgotten according to the currently input and previously output results. Through the sigmoid neural layer, the input can be compressed to the (0, 1) interval. (3) ft=σ(Wf⋅⌊ht−1, Xt⌋+bf {f_t} = \sigma ({W_f} \cdot \left\lfloor {{h_{t - 1}},\;{X_t}} \right\rfloor + {b_f} (4) Ct˜=tanh(Wc⋅⌊ht−1, Xt⌋+bc) \widetilde {{C_t}} = tanh({W_c} \cdot \left\lfloor {{h_{t - 1}},\;{X_t}} \right\rfloor + {b_c})

Input gate: the neural layer that calculates the output value at the current time. The integration vector uses the sigmoid function to extract the output layer, calculates the output of the previous time and the current time, and then maps the interval through the tanh activation function. (5) I˙t=σ(Wi⋅⌊ht−1, Xt⌋+bi {\dot I_t} = \sigma ({W_i} \cdot \left\lfloor {{h_{t - 1}},\;{X_t}} \right\rfloor + {b_i}

Output gate: it determines what to output based on the content saved in the cell state. Similar to the input gate, the output of the output gate to the content also needs to be determined by using the sigmoid activation function, and then the tanh activation function is used to process the content of the cell state. Finally, multiplying the two results is the final result that we need. (6) Ot=σ(W0⋅⌊ht−1, Xt⌋+b0 {O_t} = \sigma ({W_0} \cdot \left\lfloor {{h_{t - 1}},\;{X_t}} \right\rfloor + {b_0}

Two kinds of memory:

Long memory: to obtain the status value of the memory unit at the current time. (7) Ct=ft * Ct−1+I˙t*Ct˜ {C_t} = {f_t}\;*\;{C_{t - 1}} + {\dot I_t}*\widetilde {{C_t}}

Short memory: output of LSTM cell. (8) ht=Ot * tanh(Ct) {h_t} = {O_t}\;*\;tanh\left( {{C_t}} \right) Where h_t represents the output result of the current unit and C_t represents the state of the memory unit at the previous time.

The inspiration for this method comes from humans themselves: when our vision perceives the scene in front of us, we don’t see everything in a scene every time, but only the thing we want to see. That is when we learn that in a particular scene, the thing we want to see almost always appears in a certain part. When we learn of this characteristic through repeated exposure to similar scenes, we would spontaneously train ourselves to focus on the particular focused part only and try not to look at other parts, the objective being to improve efficiency in work.

The hierarchical attention network is shown in Figure 5.

Fig. 5

This paper uses the attention mechanism in a hierarchical attention network, puts the features extracted from vectorised words and sentences into the network layer, so as to realise different degrees of attention to text information, and uses the obtained feature vectors to realise text classification.

The LSTM algorithm has two methods: forward propagation and backward propagation. In order to make the model prediction more accurate, the LSTM model is combined with the LSTM in the positive direction and the LSTM in the reverse direction from the input layer, and then used as the next input, as indicated in Figure 6.

Fig. 6

The calculation formula is as indicated in Eqs. (9)–(11): (9) ht=f(w1xt+w2ht−1) {h_t} = f\left( {{w_1}{x_t} + {w_2}{h_{t - 1}}} \right) (10) ht′=f(w3xt+w5ht+1′) h_t^\prime = f({w_3}{x_t} + {w_5}h_{t + 1}^\prime ) (11) Ot=g​(w4ht+w6ht′) {O_t} = g\left( {{w_4}{h_t} + {w_6}h_t^\prime } \right)

In the field of sentiment classification, the CNN method has also achieved good results, but CNN involves the need to build a lot of feature engineering for improving the accuracy of sentiment classification, which will take a lot of time. When using the LSTM method, this problem is avoided, and there is no need to consider the semantic relationship between words. For a long text, LSTM can effectively connect the semantics of the context and learn sentence-level text features. Bi-LSTM can simply be understood as LSTM in two directions. The results obtained before and after can be combined to solve the problem of sentiment classification, and the effect is better in more complex sentences. In the research of sentiment classification, first, the user's comments are vectorised. LSTM neural network will selectively retain the information affecting the neural network through the unique gate structure, and update the cell state in real-time. For example, when predicting the text published by ‘In my point of view, The two singers performed badly, But listen more deeply and find that it's not as bad as that. It's intriguing’, the LSTM network will turn the previous dissatisfaction with the singer's attitude into satisfaction through the forgetting gate of its cell unit. The analysis result of user comment text finally uses the sigmoid function to display the output result as 1 (positive) or 0 (negative).

This experiment uses the data set sentiment140 (user comments provided by Twitter), which has 800,000 positive and 800,000 negative sentiment data elements. The model is trained and tested based on this data set. The obtained data sets were randomly divided into 1,280,000 training sets and 320,000 test sets based on the ratio of 4:1. Each data element is a user's English text comment. The model is obtained from the training set, and the accuracy is obtained from the test set.

The results of text sentiment analysis method based on LSTM are shown in Figure 10.

Fig. 10

The results of text sentiment analysis method based on Bi-LSTM are shown in Figure 11.

Fig. 11

In the field of sentiment classification, the CNN method has also achieved good results, but CNN involves the need to build a lot of feature engineering in order to improve the accuracy of sentiment classification, which will consume a lot of time. When using the LSTM method, this problem is avoided, and there is no need to consider the semantic relationship between words. For long text, LSTM can effectively connect the semantics of the context and learn sentence-level text features. Bi-LSTM can simply be understood as LSTM in two directions. The results obtained before and after can be combined to solve the problem of sentiment classification, and the effect is better in more complex sentences. In the research of text sentiment classification, first, the user's comments are vectorised. LSTM neural network will selectively retain the information affecting the neural network through the unique gate structure, and update the cell state in real-time. For example, when predicting the text published by the user ‘in my point of view, the two singers performed badly, but listen more deeply and find that it's not as bad as that. It's intriguing’, the LSTM network will turn the previous dissatisfaction with the singer's attitude into satisfaction through the forgetting gate of its cell unit. The analysis result of user comment text finally uses the sigmoid function to display the output result as 1 (positive) or 0 (negative).

The experimental results show that with the increase of model training times, the accuracy of the model increases steadily and the loss rate decreases gradually, as shown in Figure 12. The accuracy of test set and training set is also increasing, and the loss rate is decreasing. In the sentiment analysis of user comments, when the same text is input, both can accurately predict whether the text encompasses a positive or negative sentiment. However, when looking at the more accurate probability value of the model, the prediction accuracy of the Bi-LSTM-based method is up to about 76.8%, and the prediction accuracy of the LSTM-based method is up to about 75.4%. In some texts, the prediction accuracy of the Bi-LSTM-based method is much higher than that of the LSTM-based method. Therefore, the Bi-LSTM-based method has a higher prediction accuracy of sentiment analysis than the LSTM-based method in dataset Sentiment140.

Fig. 12