Research on Pilots ’ Mental Workload Classification in Simulated Flight
Publicado en línea: 31 may 2023
Páginas: 75 - 82
DOI: https://doi.org/10.2478/ijanmc-2023-0048
Palabras clave
© 2023 Jinna Xue et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
With the emergence and rapid development of the Internet, people’s load on cognitive resources has also increased, and the demand for mental workload optimization has become increasingly prominent [1]. Studies have shown that too high and too low cognitive load will have a negative impact on job performance [2]. Pilots need to obtain more instrument data information during driving, and combine it with the working environment for comprehensive analysis, so as to make corresponding decisions quickly and accurately [3]. In the relevant operation process requires a high degree of concentration, may produce large mental labor [4], will not only affect the work efficiency, and even cause serious accidents. Thus, in human-computer interaction, mental labor gradually occupy the advantage. The intensity of mental labor can be measured by mental workload, which is mainly generated in projects where operators use limited mental resources for task processing [5]. Appropriate mental workload helps the operator to maintain good task performance, while too high or too low mental workload will adversely affect the performance in the operation.
According to the aviation accident research report, 60 % to 90 % of flight accidents occur in flights where the pilot ’s mental workload intensity is too high and the stress task level is high [6]. Complex human-machine interaction operations such as air defense missiles, medical rescue and efficient driving are often accompanied by high-load human-machine interaction tasks [7]. For example, in the operation of missile weapons, it is necessary to achieve accurate and fast analysis of space intelligence such as the course and speed of combat targets, and to quickly realize the assessment of the threat of combat targets for the implementation of complex actions such as target tracking and attack [8-10]. Because there is a large degree of human-computer interaction tasks in this process. The high-intensity operation process can easily cause the mental operator to fall into an overload state, which greatly reduces the safety and reliability of the operator ’s operation process and easily causes safety accidents. Therefore, the study of mental workload is of great significance to the safety and reliability of human-computer interaction.
The concept of mental workload was first proposed in the 1980 s. Scholars have mainly conducted in-depth research and discussion on the causes, internal mechanisms, and measurement methods of mental workload [11]. In the past forty years, researchers have continued to pay attention to the mental workload of operators. The physiological measurement method has greater advantages in terms of sensitivity to changes in brain load and digital quantitative analysis, but there are greater difficulties in terms of post-processing of data and other aspects, a certain knowledge base is required for data analysis, and the highly sensitive measurement method is susceptible to interference from external factors. In response to these difficulties, some scholars have proposed the use of eye-movement metrics for the assessment of brain load [12-14]. Liu Yan [15] et al. conducted a correlation analysis of pilots’ brain load in terms of typical issues such as fatigue and sleep, and obtained from the experimental results that the correlation degree coefficient between brain load and multidimensional fatigue measurements was around 0.5, while the correlation degree coefficient between brain load and Pittsburgh sleep quality was as high as 0.59. From the results of this study, it can be obtained that the degree of pilots’ brain load is influenced by the degree of fatigue and sleep. Wang Lei et al. from Civil Aviation University of China [16] conducted a study on the brain load characteristics of pilots based on flight task context routes, in which they used different task difficulties to carry out relevant studies on the brain load characteristics of pilots. The experimental results are of great reference value for the study of pilot brain load classification methods. In this paper, we will draw on the relevant pilot brain load characteristics to investigate pilot brain load classification methods.
The experiment was conducted under a simulated flight platform and eight male graduate student volunteers aged 22-30 years were recruited as subjects for data collection. The subjects were all in good health, had normal or corrected vision, were right-handed and had good sleep conditions. The subjects were familiarised with the details of the experimental operation and related precautions before the simulation. When conducting the experiments, only the subjects were left alone for each trial in order to avoid interference from the outside environment.
The equipment for this pilot mental workload classification study contains a DELL computer, three high definition displays (2560 x 1440 resolution), EEG signal synchronisation acquisition equipment and the test deployment platform is shown in Figure 1. The flight simulation platform contains a monitor, flight joystick and mainframe. The flight simulation was conducted using the DCS World digital battlefield game, using a free to fly Su-25T fighter aircraft designed and manufactured by Sukhoi in Russia.
Figure 1.
Simulated flight environment experimental platform
The experimental equipment used in this experiment was the ErgoLAB human-computer environment synchronous platform EEG measurement system provided by Beijing Jinfa Technology Co. The system adopts the international standard 10-20 standard lead semi-dry electrohydraulic stage with an output bit 32-lead EEG signal; a wet sponge is used as the conduction medium, as shown in Figure 2(a).
Figure 2.
ErgoLAB EEG device
In this simulated flight experiment task, each subject performed three main simulated flight experiments containing three mental-load flight tasks: control, low load and high load. The cockpit dial information that needed to be monitored during the simulated flight corresponding to the different difficulty tasks during the simulated flight was shown in Table 1. The flight information contained in the cockpit interface [17] includes indication of airspeed table, pitch scale information, no altimeter information, heading angle, roll angle, direction pod, landing gear status and engine status. During a flight task, the subject is required to respond accurately and quickly to the information presented in the interface.
INFORMATION ON THE DIALS TO BE MONITORED FOR DIFFERENT DIFFICULTY TASKS
| High loads | Low loads | Control group |
|---|---|---|
| Indicated airspeed meter | Indicated airspeed meter | — |
| Pitching scale information | Pitching scale information | — |
| Altimeter information | Altimeter information | — |
| Directional angle information | — | — |
| Rolling corners information | — | — |
| Status of the steering compartment | — | — |
| Landing gear conditions | — | — |
| Engine status | — | — |
For different subjects, the experiments are conducted one by one. Each subject is required to perform a control, low and high load flight for each experiment. In the same flight environment, the operator is prompted with different difficulty tasks based on the flight heading status and is required to quickly and accurately monitor the corresponding dial information.
The NASA-TLX Subjective Rating Scale [18] was used in the experiment. The scale of [0,100] was used to indicate the scale’s score range to rate the subject’s level of mental workload on six dimensions, with higher scores indicating higher levels of workload.
During the experiment, simultaneous acquisition of EEG signals was performed using EEG acquisition equipment. At the end of each set of simulated flight experiments, subjects are required to complete the NASA-TLX measurement form, which is used to evaluate the subjective load of the subjects.
In this study, a modified long and short term memory network was used to classify the EEG signals for mental workload research, and the flight mental workload was studied and analysed by combining subjective evaluation methods.
During the experimental process, the hardware equipment will inevitably be disturbed by external factors. Therefore, the raw EEG signal needs to be pre-processed before the mental workload analysis is carried out, and the processing flow is shown in Figure 3.
Figure 3.
Flow chart of data processing
Related studies have shown that most EEG signals are concentrated between the range of 0.05-50Hz, while event-related desynchronisation and event-related synchronisation (ERD/ERS) patterns are mainly manifested in μ rhythms (8-13Hz) and β beats (14-30Hz), so this study used Butterworth filters for 50Hz IDF trapping and 0.05-30Hz bandpass filtering, and Butter The Butterworth band-pass filtering removed the fine burrs of interference from the original data [19]. A comparison of the data before and after processing is shown in Figure 4.
Figure 4.
Comparison of EEG waveforms before and after data processing
According to recent research in cognition, psychology and psychiatry, EEG information about mental activity can be divided into five bands: delta ( δ ), theta ( θ ), alpha ( α ), beta (β ) and gamma ( γ ), and the frequency range of each band is shown in Table 2.
TABLE OF FREQUENCY BANDS OF EEG SIGNALS
| Frequency band range(Hz) | Band names |
|---|---|
| 1-4 | delta (δ) |
| 4-8 | theta (θ) |
| 8-13 | alpha (α) |
| 13-30 | beta (β) |
| >30 | gamma (γ) |
In this paper, the discrete Fourier transform is used for EEG signal feature extraction [20]. The power spectrum was estimated mainly using the Welch method with the power spectral densities of the
Where,
The network model proposed in this paper is shown in Figure 5. The model is a new input convolutional neural network in front of the long and short-term memory network for local feature extraction of the EEG signal. The extracted local features are then fed into the long and short-term memory network for training. The loss function for model training is L1-loss and its expression is:
Figure 5.
Network model diagram
The Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) [21]. The LSTM contains a three-part structure of input gates, forgetting gates and output gates, which is shown in Figure 6 and mainly outputs (0,1) through the sigmoid activation function values between.
Figure 6.
LSTM structure diagram
The role of the forgetting gate is to control the discard rate of the delivered information, as implemented in equation (3).
Where
The input gate determines at what percentage of the current moment’s input information is fed into the memory information stream, and is calculated similarly to the forgetting gate, as shown in equation (4).
The output gate is mainly used to control the amount of information updated by the next layer of the network and is implemented as shown in equation (5).
Based on the results of the NASA-TLX subjective evaluation table obtained statistically, the sensitive assessment index scores for changes in mission difficulty can be obtained as shown in Figure 7. According to the ANOVA results, the NASA-TLX subjective evaluation scores show an extremely significant difference (p<0.01) and become larger as the difficulty of the mission increases and the load level increases.
Figure 7.
NASA-TLX evaluation results
Based on the results of the NASA-TLX subjective evaluation table obtained statistically, the sensitive assessment index scores for changes in mission difficulty can be obtained as shown in Figure 7. According to the ANOVA results, the NASA-TLX subjective evaluation scores show an extremely significant difference (p<0.01) and become larger as the difficulty of the mission increases and the load level increases.
This paper presents a study related to the use of a modified long and short term memory network for mental workload classification methods. In conducting the model training process, the pre-processed data was used as the input data for the model, where 80% of the data was used to conduct the model training and 20% of the data was used to conduct the test. Details of the parameter settings of the model are shown in Table 3.
MODEL TRAINING PARAMETERS
| Name of the layer | Hyperparameters | Input vector | Output vector |
|---|---|---|---|
| EEG_Input | None | (256,1) | (256,1) |
| Conv1D | Kernel size=5 | ||
| stride = 2 | (256,1) | (128,64) | |
| Conv2D | Kernel size=3 | ||
| stride = 2 | (128,64) | (64,128) | |
| Reshape | None | (64,128) | (8192,1) |
| Max Pooling | pool size=128 | (8192,1) | (64,1) |
| LSTM | units=32 | (64,1) | (32,1) |
| Dense | units=64 | (32,1) | (64,1) |
| Dense | units=32 | (64,1) | (32,1) |
The number of times the model was trained was set to 500 and the learning rate was set to 0.001. The experimental results showed that the training accuracy could basically reach over 90% around the 120th epoch. In the first 20 epochs and 70~80 epochs, the discriminant accuracy improved faster, and the experimental result accuracy is shown in Figure 8; the loss value converged more rapidly, and the experimental loss rate is shown in Figure 9. In this model training, the final accuracy rate can reach 94.9%. Compared with other traditional methods, this experimental model has greater accuracy.
Figure 8.
Training and test set accuracy
Figure 9.
Loss rate for training and test sets
Piloting in aviation is a major hazard and pilots are under great stress during the flight. The study of the pilot’s mental workload is conducive to a better assessment of the pilot’s driving state, as well as to the effective prediction of phenomena that exceed the mental workload, which is conducive to the rational design of the human-computer interaction system in flight driving, thus reducing the phenomenon of high or low mental workload during flight driving and ensuring the safety of flight driving.
This paper uses a combination of convolutional neural networks and long and short-term memory networks to classify mental workload with an accuracy of 94.9%. The results of the analysis show that there is a positive correlation between the difficulty of the flight task and the subjective evaluation score. In summary, this study has some reference value for the classification of pilot mental workload.