Research on Fatigue Classification of Flight Simulation Training

International Civil Aviation Organization (ICAO) statistics on global planned commercial flight accidents and casualties in the past ten years show that from 2013 to 2017, the annual number of flight accidents has not changed much, but the number of casualties has been there have been large fluctuations, and the overall number of casualties remains high. In addition, once an accident occurs, the direct loss caused is the production and manufacturing cost of an aircraft (the average value of Boeing’s aircraft supplies is approximately $90 million), plus compensation for accident losses. Although the flight accident rate has been on a downward trend in recent decades, the injuries and losses caused by airplane accidents have not changed much. At the same time, every flight accident will cause fatal injuries to every family and indirectly cause national losses. Therefore, aviation safety issues need to be treated strictly for countries in all regions of the world.

According to the flight accident statistics of the Federal Aviation Administration (FAA) and NASA, only 12% of flight accidents are caused by problems with the aircraft itself, and more than 73% of accidents involve human factors. 67% of aircraft accidents caused by mistakes of the aircraft, the most important factor is the pilot's operation error, which accounts for about 51% of the total number of air crashes. The main reason for the pilot's operation error is that the driver is in a state of fatigue and his driving is alert. Degree drops. According to the statistics of road traffic accidents in my country, 90% of traffic accidents are caused by the driver’s human factors, which are mainly due to the driver’s dangerous driving state, such as distracted driving, fatigue driving, etc.

The Current methods for detecting pilot alertness go in three directions, namely aircraft-based behavioural monitoring, pilot behavioural recording and pilot physiological signal measurement. Of these, the first two methods are more influenced by the external environment such as the aircraft model and driving environment, while the latter method depends only on the subject conditions; therefore, it shows a higher ability to detect driver drowsiness. Measurements of physiological signals include neuronal electrical activity from electroencephalography (EEG), eye movements from electrooculography (EOG), heart rate from electrocardiography (ECG), muscle activity from electromyography (EMG) and tissue oxygenation from near infrared spectroscopy (NIRS) [1]. However, of these signals, the EEG signal, which is less likely to be influenced by individual characteristics, has been widely used as the ‘gold standard’ for fatigue detection and has proved to be a promising method for studying drowsiness and changes in driver alertness.

EEG is the overall response of brain nerve cell electrophysiological activity on the scalp surface or cerebral cortex [2]. According to its frequency, it can be divided into 5 different bands: (1) δ wave (1~4 Hz), which generally only appears when adults are asleep; (2) θ wave (4~8 Hz), which mainly occurs in sleep State; (3) α wave (8~14 Hz), which generally occurs in a relaxed state; (4) β wave (14~30 Hz), the increase in the power spectrum of the β wave is closely related to the increase in alertness. (5) Gamma wave (30–49 Hz).

As early as the 1980s, studies on the correlation between EEG and brain fatigue have been carried out abroad. Relevant studies have shown that EEG is very sensitive to fluctuations in vigilance. EEG will change significantly with changes in vigilance. EEG It can predict the decline of brain performance caused by continuous mental work [3]; in the 1990s, EEG research was further deepened, and people began to pay attention to the changes in various bands of EEG during brain fatigue. Research found the correlation between reduced human alertness and fatigue It will be concretely reflected in different wave bands of EEG, among which the changes of theta wave and alpha wave are particularly obvious [4], which is specifically manifested in the power spectrum of theta wave and alpha wave when people are in a state of fatigue; enter 21 In the century, the research on the different bands of EEG in the state of brain fatigue is more detailed, the θ, α and β frequency bands in EEG and the combined parameters of different frequency bands. For example, Jap et al. [5] conducted a more comprehensive experiment. The four EEG bands of δ, θ, α, β and (θ+α)/β, α/β, (θ+ The evaluation of the four parameters of α)/(α+β) and θ/β showed that during the transition from non-fatigue state to fatigue state, the activities of δ wave and θ wave were relatively stable, and the activity of α wave decreased slightly. The β wave activity is significantly reduced; the values of the four combination parameters have increased, among which the increase in (θ+α)/β is more obvious, and the value of (θ+α)/β is also more obvious under different fatigue levels the difference. By combining EEG signals of different bands for analysis, and then proposing suitable combination parameters, not only can the respective characteristics of different bands of EEG signals be fully utilized, but also the detection results can be more accurate and comprehensive through the combination of parameters.

Many domestic scholars have also conducted research on the alertness of drivers by EEG. In addition to the above four combined EEG features, there are also time-domain analysis and frequency-domain analysis of brain waves, power spectral density (PSD), and entropy. Carry out the analysis and evaluation of the driver's alertness from an equal angle. EEG is also an important parameter in human factors engineering. EEG's brain fatigue detection is also widely used in human factors engineering. For example, using the EEG fatigue detection method to evaluate and guide the professional training and psychological adjustment of pilots, it was found that the level of positive emotions of pilots has been improved. Similarly, the brain fatigue detection using EEG can also study the effect of brain fatigue on selective visual attention, and the results show that mental fatigue has a negative impact on the ability of selective visual attention.

At present, the use of EEG can be used to determine the degree of brain fatigue and brain fatigue. With the development of high-throughput EEG technology and the intelligence of EEG data analysis, traceability analysis of high-throughput EEG data is expected to be used for brain fatigue location analysis; on the other hand, the fusion of synchronous EEG and functional magnetic resonance imaging technology also provides technical support for brain fatigue location analysis. In short, EEG-based brain fatigue detection will develop in the direction of quantitative, precise, and accurate positioning, and the fatigue detection ability and credibility of drivers will also continue to improve.

Carlo Matteucci et al [13]. First obtained the muscle nerve electrical signal with a galvanometer in 1881 and established the concept of neurophysiology. In the following nearly a hundred years, people gradually clarified the collection methods and standards of bioelectric signals. The non-invasive collection methods that have been widely used in the field of EEG signals include EEG, magnetic resonance imaging, near-infrared spectroscopy, and there are four kinds of magnetic EEG [14]. Among them, the multi-channel electrode EEG method, which integrates high time resolution, low cost, and non-invasive safety, is the most widely used. Thanks to the continuous advancement of technology, the 10–20 standard lead, 10–10 standard lead and 10-5 standard lead established by the American Society of Clinical Neurophysiology are the most common in clinical trials [15]. The three standard system guides are extensions of each other and keep the same overlap in the naming rules. This simplifies the EEG signal research process and reduces the difficulty of technical communication. It also clears the obstacles for the electrode naming rules for this study. Specific electrode naming and the spatial coordinates can be queried on the website of the American Society of Clinical Neurophysiology, and will not be listed in detail here. In this project, the electrode position recommended by the 10–10 standard lead system will be used as a benchmark to start the experiment.

The EEG signal is a sign of neural activity. The neural activity generated by any part of the human body will be more or less reflected in the EEG signal [16], and the research needs to focus on the “event potential”, that is, the human brain because of a certain One or some physiological electrical signals generated by certain activities, so choosing a suitable reference electrode according to different research focuses will greatly reduce the research workload. The research of Lei Xu et al. [17, 18] showed that the reference electrode is the key to the study of EEG and event-related potentials. Since Yao et al. proposed the reference electrode standardization technology in 2001, REST has quickly become the first choice for most EEG research models. In addition, in some cases, the full electrode average can also be considered to have the same effect as REST. Essl et al. suggested choosing FCz electrode as a reference electrode when the results of the study are not clear. The consistency based on FCz electrode is higher than the consistency based on non-cranial reference, and their research has become part of REST. Yao Dezhong et al. studied the influence of different reference electrodes on spectrum mapping. REST technology aims to build a bridge between traditional reference electrodes (such as scalp or average reference) and theoretical zero reference. The reference point at infinity has a theoretical neutral potential and is regarded as an approximate zero potential point in REST.

EEG signal is a 5–100μV low-frequency bioelectric signal, which needs to be amplified before it can be displayed and processed. An important operation in signal processing and interpretation is filtering. The main function of filtering is to remove interference signals from EEG signal data. Especially for high-frequency signals caused by the external environment, the filtering used in EEG signal processing is mainly divided into high-pass filtering, low-pass filtering and notch filtering. The filtered data can be analyzed for characteristics. There are two main characteristics of EEG data: spatial characteristics and time-frequency characteristics. High-pass filtering aims to pass signals above a limited frequency without attenuation, while blocking and attenuating signals below the limited frequency. Since the EEG is a signal of about 30 Hz, and the frequency above 50 Hz is only involved in the medical diagnosis of epilepsy and human brain physiology, high-pass filtering is rarely used in EEG signals, but some scholars choose to do it. The high-pass filter of 0.1~0.7Hz is designed to remove frequency components with extremely low interference such as breathing. If there is a problem of baseline drift in the signal, Alste et al. conducted a study on ECG and suggested using high-pass filtering to deal with such problems. Although high-pass filtering may be one of the means to solve the baseline drift, Acunzo et al. found that high-pass filtering can cause early ERP and ERF system deviations. High-pass filtering is used cautiously when dealing with the model of EEG and ERP signal fusion.

Low-pass filtering allows signals with frequencies below a certain range to pass, and signals above the critical frequency are blocked and attenuated. Because the EEG signal acquisition instrument is sensitive to weak electrical signals, and the frequency of the mains power in my country is around 50Hz, although there is still a certain frequency space below 30Hz from the target, low-pass filtering must be performed to reduce the impact of the mains signal on the data. This operation is also one of the normal operations in EEG signal processing.

McFarland et al. in 1997 proposed a spatial filter selection based on EEG, by selecting different filters to process the signal to obtain a clear EEG signal. When studying EEG signals, Higashi et al. found that the spatial filter based on the common space pattern method of electrode weights is very effective in the classification of EEG signals based on moving images, but the existing methods have certain limitations. For this reason, a discriminative filter bank is proposed to extract the bands related to the brain activity of moving images.

The time-frequency domain method is also a common EEG signal research method. Hjorh, Salinsky and Valdes discussed the reliability of EEG frequency domain analysis. An important step in time-frequency analysis is to convert time-domain signals into frequency-domain signals. If the signal is statistically stable, or there is a fixed law, then the finite length signal can be transformed into a frequency form by using a linear transformation.

EEG data analysis involves a variety of signal processing techniques, including but not limited to signal acquisition, preprocessing, and feature extraction. There are also a variety of methods that are widely used in data classification, such as KNN based on sample feature distance and VC theory. Linear SVM and models based on convolutional neural networks. This chapter will respectively introduce the key technologies of the above three fields involved in this research.

EEG signal processing is mainly composed of signal acquisition, conversion reference, filtering, artifact removal, segmentation, independent component analysis and other operations. In addition, you can also choose whether to downsample the collected data according to the actual situation, but you need to pay attention if it is down-sampling, it may be necessary to perform linear or non-linear interpolation to complement the disappeared features. This article mainly extracts the four features of EEG δ, θ, α, β for fatigue classification.

C.

Convolutional Neural Network

According to the characteristics of EEG, a two-dimensional convolutional neural network is used in the design of the neural network. In the experiment, the data set was randomly divided into training set and test set at a ratio of 8:2, and then a neural network structure was established for training on EEG features. The training iterations were 50 times, and the learning rate was set to 0.001. Through the analysis of the experimental results, the loss rate and accuracy curves of the training set and test set of EEG feature training are shown in Figure 3 and Figure 4. In the traditional fatigue detection method SVM, the average accuracy of KNN is 86%and 72% respectively. Compared with the traditional fatigue detection method, the convolutional neural network method has indeed improved a lot, especially the convolutional neural network method proposed in this paper can achieve 98%.

Figure 3.

Figure 4.

The detection of fatigue state is widely used in life, and the judgment of fatigue state is helpful to the safe operation of the operator, can reduce the occurrence of accidents, and protect the physical and mental health of the operator. In order to further verify the effectiveness of the method in identifying fatigue classification, the method in this paper is compared with traditional K nearest neighbors, support vector machines and the current more popular deep learning methods. The classification and recognition accuracy rates of KNN, SVM and CNN reached 72% respectively., 86% and 98%, the accuracy of the method in this paper reaches 98%, which better realizes the operator’s fatigue classification. Carry out operator fatigue classification, the classification accuracy rate reaches 98%, and the fatigue state classification is well realized. At the same time, the complex feature extraction process in traditional algorithms is avoided, which is beneficial to real-time and accurate detection of operator fatigue. In addition, applying the same model to different subjects, the classification accuracy rate of each subject's fatigue state exceeds 93%, which can better eliminate the influence of individual differences.

摘要—疲劳是影响现代飞行安全的重要因素. 很容易导 致飞行员操作能力下降, 误判和飞行幻觉. 而且, 它甚 至可能导致严重的飞行事故. 本文采用可穿戴无线生理 设备在模拟飞行实验中获取飞行员心电图数据. 生物电 信号比图像信息具有更高的可靠性, 不易受外界环境（如 拍摄角度, 光照强度等）的影响. 另一方面, 神经网络 已广泛用于各种分类和回归任务. 本研究将脑电图采集 在驾驶飞行模拟器中, 经过简单的滤波和预处理后, 将 时域数据直接发送到卷积神经网络, 无需额外的特征提 取操作. 我们发现卷积神经网络可以有效放大时域数据 的波动细节, 训练飞行员疲劳状态识别模型. 结果表明, 卷积神经网络模型的识别准确率达到了 98%, 分别比传 统的 k 近邻分类算法（KNN）和支持向量机（SVM）模 型提高了 26% 和 12%. 本文建立的基于卷积神经网络的 识别模型适用于飞行员疲劳状态的识别. 这对于减少飞 行员疲劳造成的飞行事故具有重要的现实意义, 为飞行 员疲劳风险管理和智能飞机自动驾驶系统的发展提供了 理论依据.

关键词-疲劳分类; 脑电; 卷积网络; 神经网络

2.

实验

2.1

实验设计

模拟飞行驾驶与真实驾驶的难度和危险程度 存在一定差距｡为减少新老学员对驾驶任务控 制能力的影响,保证测试数据的客观性,选取 20 名研究生,年龄 24–28 岁(平均年龄 26.8 岁),身体 健康,都是右撇子;没有驾驶经验;测试期间没有 服用任何药物;测试前 24 小时未食用酒精食物, 测试前 12 小时未食用含咖啡因的饮料｡前 1 小 时没有吃东西和剧烈运动;为避免测试周期的 影响,测试在与人体生理周期相近的周期内完 成;考试前 1 天,对受试者进行驾驶模拟器操作 20 分钟的培训,并尽量保证样本正确､驾驶水平 的相同性和平等性｡参与者被告知实验的具体 内容,以确保他们充分了解在研究期间收集的 生理数据将如何被使用｡每个参与者都愿意参 与实验;实验数据必须剔除个人身份信息,只保 留 对 实 验 有 特 定 影 响 的 数 据 ｡研 究 在 中 午 (12:00–14:00)进行,室内温度控制在 25±2℃[6], 室内湿度为 45±10%[7]｡要求所有参与者在实验 前 5 小时内进行适度的脑力劳动,以降低神经兴 奋性｡他们不允许参加任何形式的体力劳动,以 防止血压和心率的变化｡每个参与者在实验前 都有 8–9 小时的睡眠时间｡观察者被设置为记录 实验者的状态,包括眼睛是否有红血丝和眨眼 频率的变化[8]｡神经网络模型采用 Tensorflow GPU 2.4.0 框架,CUDA 版本为 11.0,cuDNN 版本 为 8.0｡我们使用 4 块 NVIDIA Titan V 显卡来加 速训练过程｡

2.2

实验流程

实验刺激呈现在 31.5 英寸桌面曲面显示器上｡屏幕上的互动对象是由 Dovetail Games 发行､ 航空专家 Jane Whittaker 开发的《微软飞行模拟 器 X:Steam 版》游戏｡这款游戏可以让非职业玩 家在遇到紧急情况时感受到飞行员的紧张情绪 ｡画面和逼真程度达到了巅峰,现实飞行中遇到 的各种元素,如空气动力学､天气､地理环境､飞 行控制系统､飞行电子系统､作战飞行武器系统､ 地面飞行制导等在计算机中进行综合模拟,通 过外部硬件设备进行飞行模拟控制和飞行感官 反馈,反馈飞行员上一年的疲劳表现｡以眼动､视 线等为基础,完成飞行员训练科目(起飞､着陆) 的脑电采集与分析,识别飞行员训练过程中的 情绪特征,完成多维通道数据融合,构建飞行员 控制响应时间和注意力分布的数学模型监控和 评估飞行训练的效果并进行控制实验｡进入模 拟控制环境前,被试需佩戴脑电设备,确保设备 与软件连接进行电阻测量(Ergo 软件显示端口 为绿色), Ergo 软件会显示端口的连接状态.有四 种:1.绿色:端口连接正常,信号稳定 2.橙色:端口 连接正常,信号不稳定 3.红色:端口连接正常,信 号弱 4.灰色:实验前端口无信号,需确保设备与 软件连接正常,测量电阻(所有端口均为绿色),并 眨眼闭眼确认设备正在接收眼球一般｡测试对 象进入模拟飞行环境后,需要在半小时内不断 与实验材料交互并逆行,完成固定翼飞机的起 降,绕场飞行五圈,继续飞行指定的路线｡在正式 测试过程中,受试者需要进入模拟机动场景(如 图 1),飞行模拟平台模拟驾驶的实验任务｡

图 1

图 2

任务执行过程中,被试视野范围内会随机出 现弹窗,被试需要将视点移至弹窗关闭窗口｡从 弹出窗口出现到关闭的时间间隔就是反应时间 ｡根据响应时间,研究团队在弹出窗口出现前 10 秒内为脑电信号开发了一个心理标签(疲劳/心 理)｡一组实验设置半小时｡因为脑电帽会影响被 试,时间过长会增加被试眼高,造成极度不适｡因 此,本课题将一组实验分为两部分,具体时间表 如表 1 所示｡

表 1

实验计划表

Participant ID	001	002	003
Experiment 101	9:00–9:30	10:00–10:30	11:00–11:30
Experiment 102	13:30–14:00	14:30–15:00	15:30–16:00

在完成任务期间,限制受试者的身体和头部 进行更剧烈的运动｡而且,在实验开始前,记录仪 会告诉被试,闭眼的行为应该是自发的,而不是 刻意的,比如在感到困倦或不舒服的时候让眼 睛闭上｡由于对人身安全没有实际威胁,因此对 象在疲劳时会顺其自然｡这些情况在真实的操 作环境中是很难获得的｡这有助于我们研究人 体疲劳时生理信号变化的特征｡

4.

研究方法

EEG 数据分析涉及多种信号处理技术,包括 但不限于信号采集､预处理和特征提取｡还有多 种方法被广泛应用于数据分类,如基于样本特 征距离的 KNN 和 VC 理论｡线性 SVM 和基于卷 积神经网络的模型｡本章将分别介绍本研究涉 及的上述三个领域的关键技术｡

脑电信号处理主要由信号采集､转换参考､滤 波､去伪影､分割､独立分量分析等操作组成｡此 外,还可以根据实际情况选择是否对采集的数 据进行下采样,但需要注意的是,如果是下采样, 可能需要进行线性或非线性插值来补充消失的 特征｡本文主要提取 EEG 的 δ､θ､α､β 四个特征进 行疲劳分类｡

4.1

K 最近邻法

K 最近邻(KNN)算法是常用的分类算法之一｡当 对 数 据 分 布 知 之 甚 少 或 没 有 先 验 知 识 时,KNN 应该是首选方法｡Cover 等人[9]｡明确 了 KNN 算法的分类误差上限是贝叶斯分类误 差的两倍｡该算法旨在计算未知样本与已知样 本组之间的特征距离,并根据距离推断未知样 本的类别｡常见的距离有欧几里得距离､闵可夫 斯基距离､曼哈顿距离､切比雪夫距离等｡

欧式距离是最常用的测量方法,它测量多维 空间中点之间的绝对距离,定义如下｡ (1) d(x, y)=∑i=1n(xi−yi)2 {\rm{d}}\left( {{\rm{x}},\,{\rm{y}}} \right) = \sqrt {\sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {{{\left( {{{\rm{x}}_{\rm{i}}} - {{\rm{y}}_{\rm{i}}}} \right)}^2}}}

由于欧氏距离是根据各维度特征的绝对值计 算的,所以使用欧氏距离的前提是要保证指标 的维度具有相同的维度｡不同的维度可能会导 致欧式距离无效｡心灵距离是欧几里得距离｡概 括来说,下式中的当前 p=2 就是欧几里得距离｡ (2) d(x, y)=(∑i=1n|xi−yi|p)1p {\rm{d}}\left( {{\rm{x}},\,{\rm{y}}} \right) = {\left( {\sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {{{\left| {{{\rm{x}}_{\rm{i}}} - {{\rm{y}}_{\rm{i}}}} \right|}^{\rm{p}}}}} \right)^{{\raise0.5ex\hbox{$\scriptstyle 1$}\kern-0.1em/\kern-0.15em\lower0.25ex\hbox{$\scriptstyle {\rm{p}}$}}}}

曼哈顿距离源自城市街区距离. 多维距离求 和的结果, 即公式中 p=1 时, 得到曼哈顿距离. (3) d(x, y)=∑i=1n|xi−yi| {\rm{d}}\left( {{\rm{x}},\,{\rm{y}}} \right) = \sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {\left| {{{\rm{x}}_{\rm{i}}} - {{\rm{y}}_{\rm{i}}}} \right|}

此外, 特征转换还可以在一定程度上提高模 型的准确性［１０］. 常用的特征变换包括标准 化和模糊化. 标准化消除了同一维度状态下不 同尺度的影响, 而模糊化则利用特征值的不确 定性来提高性能. ＥＥＧ数据分析领域的特征 模糊化可以在ＫＮＮ中表现出更好的性能. 杨 等人［１］. 对测距的效果进行了详细的讨论, 得出结论：与单纯使用欧式距离相比, 根据实 际情况设计测距大大提高ＫＮＮ分类的准确率.

4.2

支持向量机

支持向量机是机器学习中常用的工具之一｡ 与深度神经网络相比,支持向量机特别擅长处 理特征维度大于样本数量的情况｡在小样本领 域,支持向量机优于深度神经网络｡选择[13]｡线 性支持向量机旨在找到一个远离所有类型样本 的超平面｡当样本有随机扰动时,远离样本的超 平面对扰动有很强的容忍度,使得 SVM 不容易 过拟合组合｡线性支持向量机的本质是一个凸 二次规划问题｡ (4) argmaxw,b(mini2‖w‖|wTxi+b|) {\rm argmax}_{w,b}\left( {{{\min}_i}{2 \over {\left\| w \right\|}}\left| {{w^T}{x_i} + b} \right|} \right)

其中,w,b 为超平面的权重和偏移量的向量｡ 此时 SVM 的学习目标是找到一组合适的 w,b 值, 从而解决规划问题｡

4.3

神经网络

根据脑电图的特点, 在神经网络的设计中采 用了二维卷积神经网络. 实验中, 将数据集以 8:2 的比例随机分为训练集和测试集, 然后建立 神经网络结构进行脑电特征训练. 训练迭代次 数为 50 次, 学习率设置为 0.001. 通过对实验 结果的分析, EEG 特征训练的训练集和测试集 的损失率和准确率曲线如图 3 和图 4 所示. 在 传统的疲劳检测方法 SVM 中, KNN 的平均准 确率为 86%和 72%. 与传统的疲劳检测方法相 比, 卷积神经网络方法确实提升了很多, 尤其 是本文提出的卷积神经网络方法可以达到 98%.

图 3

图 4