Human behaviour recognition is a method to realise the computer's understanding and description of human behaviour. It is a critical technology in related fields such as video surveillance, competent nursing, and human-computer interaction. Periodic motion data sequences obtained through wearable motion capture systems often contain components, including hundreds of data points. To reduce the amount of calculation and eliminate the interference of redundant information, window processing is usually used to cut the data. The human body movement itself has continuity, and this applies for any daily human behaviour. The movement of hands, feet or other body parts presents a continuous process [1]. Most of the existing methods adopt direct feature extraction of discrete data collected by sensors, ignoring the study of the continuity of the movement data itself. This article attempts to take the discrete data continuity and then find the implicit continuous feature information that is closer to the essence of motion from the continuous data.
Human behaviour actions are divided into approximately periodic behaviours and non-periodic behaviours. Approximately periodic behaviours refer to exercise behaviours that are repeatedly performed multiple times. The daily behaviour of the human body often has periodic characteristics, and the data sequence of daily behaviour collected by sensors is processed at the same time. It also presents approximate periodic characteristics [2]. This article considers converting this discrete motion data sequence into a periodic continuous function, which provides a theoretical basis for the study of the continuity characteristics of motion data and can accurately express the periodic characteristics of motion data. Therefore, this article introduces a discrete data functional processing method, that is, functional data analysis (FDA). The basic idea is to treat the observed sample as a discrete data sequence and treat the observed data as a whole and express it to analyse the sample using continuous data processing methods. At present, FDA methods are mainly used in several economic data processing activities, and research on sports data is also involved.
After the feature extraction of the motion data, the appropriate classifier algorithm needs to be selected. The classifier methods commonly used for human behaviour recognition mainly include support vector machine (SVM), decision tree (DT), artificial neural network (ANN), and sparse classification (SRC). Among them, SVM can better solve practical problems such as small samples, non-linearity and high dimensionality. For the data types in this article, SVM is used as the classifier. The difference in the data characteristics within the period is used to classify and recognise the dynamic behaviour of the human body.
The FDA is a statistical method that has emerged in recent years. It is a development and extension of traditional statistical analysis methods. The FDA's application of exercise data research is reasonable. For an observation data sequence
SVM is a machine learning method. Its essence is a linear classifier that maximises the interval in the feature space. The learning strategy is to find a hyperplane in the feature space that maximises the sample interval, i.e. to solve a convex quadratic programming problem in the process [3]. The crucial part of SVM is to solve the following optimisation problems
The algorithm in this paper is mainly aimed at the periodic behaviour data of the human body collected by the wearable motion capture system. Since the human body motion has continuity and periodicity, it is necessary to functionalise the discrete motion data to obtain the continuous characteristics of the human body motion.
We use k-order Fourier series fitting for this periodic discrete observation data, as shown in Eq. (5). In the equation,
A single sensor is at the time
According to different types of behavioural data, different sample function matrices are obtained. The morphological curve characteristics of the corresponding functions in different sample function matrices are different. This difference in morphology is the basis for the classification of the sample set, and this difference can be reflected in one exercise cycle, as shown in Figure 1. Therefore, only one cycle of data can be extracted to distinguish different behaviours.
Selecting the starting point of the period through the element function of the matrix in Eq. (13) is essential for extracting period data. The method adopted in this paper is to calibrate the starting point of a period through the information of the derivative function of the fitting function (as shown in Figure 2) and then find a motion cycle. From the sample function matrix of Eq. (13), we select one of the element functions, abbreviated as
We choose the minor extreme point as the starting point of the cycle and achieve data alignment. From the perspective of the function, since there are multiple extreme points in the same function, the minor extreme point is selected to unify the standard so that the primary function represents the different functions that are aligned at the same starting point, thereby facilitating comparison of the data of different shapes and changing trends in a period. In a practical sense, the minimum extreme point of the vertical acceleration function of the foot is the minimum acceleration value during the movement. This point in time often corresponds to when a person's foot is raised to the highest point, and the periodic behaviour based on foot movement includes this movement process [5]. Therefore, we use this point as the starting point of the movement. All sample functions of the same action are aligned here, and periodic functions of different behaviours can be aligned at this starting point.
After the starting point of the period is calibrated, a period of data can be extracted according to the fitting function. In this paper, the fitting function is resampled to obtain approximately one period of motion data to express the curve characteristics of the function in one period. The sampling frequency, the time for an ordinary person to complete a motion cycle and the sampling interval of resampling directly determine the length of the resampling period sequence [6]. Assuming that the sampling frequency of the sensor is F, the time for an ordinary person to complete a motion cycle is
This paper uses the SVM method with the radial basis function as the kernel function for classification. For the radial basis kernel function, the parameter C in Eq. (6) and the parameter
This article uses Windows 7 64-bit operating system and Matlab software to conduct experiments. The data set was used from the wearable sensor behaviour recognition database (WARD) provided by the University of Berkeley in the United States. Sensors are bound to five parts of the wrist, waist, left ankle, and right ankle [7]. Each sensor is composed of a three-axis accelerometer and a two-axis gyroscope, i.e. composed of five measurement units. The sensor sampling frequency is 30 Hz. The data constituting the set encompasses 13 types of daily behaviours. When collecting data, 20 collectors (13 men and 7 women) must do each action five times. Therefore, the data set includes a total of 1,300 samples. This article selects 10 types of dynamic behaviours, with 1000 samples; see Table 1.
Behaviour description in WARD
1 | Walk normally | Walk forward for >10 s |
2 | Walk counterclockwise | Go counterclockwise for >10 s |
3 | Walk clockwise | Clockwise for >10 s |
4 | To the left | Turn left on the spot for >10 s |
5 | Turn right | Turn right on the spot for >10 s |
6 | Go up the stairs | Go up >10 stairs |
7 | Down the stairs | Go down >10 stairs |
8 | Jogging | Jogging lasts >10 s |
9 | Jump | Jump in place >5 times |
10 | Push wheelchair | Push the wheelchair for >10 s |
WARD, wearable sensor behaviour recognition database.
WARD uses five sensors to collect data. Each sensor has five measurement units (a three-axis accelerometer and a two-axis gyroscope can measure acceleration in three directions and angular velocity in two directions). Therefore, Eq. (13) gives the values of both N and M as 5.
When the sampling frequency F = 30 Hz, good experimental results have been obtained by taking 45 points in the sensor sequence, and the time of 1.5 s is close to the time to complete a behavioural action. In this article, the time for an ordinary person to complete a motion cycle is
Therefore, the length of the eigenvector in Eq. (16) is
Figure 4 shows the result of selecting a sample of the collector walking normally, extracting the acceleration data in the vertical direction of the sensor at the left ankle (that is, the data of a single measurement unit), and performing periodic extraction. Figures 4(a) and 4(b) show that the algorithm in this paper achieves a relatively accurate periodic extraction of the motion data function.
Figure 5 shows all samples of normal walking behaviour and upstairs behaviour. The acceleration data in the vertical direction of the left foot sensor is periodically extracted [9]. It can be seen from Figure 5 that the sample data are aligned at the uniform starting point of the cycle, the data length is approximately one cycle and the periodic data of different behaviours have morphological differences.
In the simulation experiment, we adopted the K-CV method to select test samples and training samples. During the experiment, we divided the data set into K = 20 groups. Each time, 1 group was selected as the test sample, and the remaining 19 groups were used as the training sample, giving rise to a total of 20 experiments. Experiments show that the algorithm in this paper has a recognition rate of 97.5% for periodic human behaviours based on the WARD data set. The specific results are shown in Table 2.
Confusion matrix of 10 dynamic behaviour categories
1 | 97 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 2 | 97 |
2 | 0 | 98 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 2 | 98 |
3 | 0 | 0 | 100 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 100 |
4 | 0 | 0 | 0 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | 100 |
5 | 0 | 0 | 0 | 0 | 98 | 0 | 0 | 0 | 0 | 0 | 98 |
6 | 0 | 0 | 0 | 0 | 0 | 98 | 1 | 1 | 1 | 0 | 98 |
7 | 0 | 0 | 0 | 0 | 0 | 2 | 94 | 1 | 0 | 3 | 94 |
8 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 99 | 0 | 0 | 99 |
9 | 0 | 0 | 0 | 0 | 0 | 1 | 5 | 0 | 100 | 0 | 100 |
10 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 100 | 100 |
Total recognition rate | 98.4 |
Table 3 and Figure 6 show the results of simulation experiments with different algorithms in the same environment using the WARD data set in Table 3. As shown in Table 3 and Figure 6, in terms of algorithm recognition rate, the recognition rate of the algorithm in this paper is high. It is similar to traditional algorithms [10]. Therefore, it cannot be compared with traditional classification methods in terms of algorithm performance.
Comparison of algorithm results based on WARD
Traditional algorithm 1 | 200 | Time-frequency domain characteristics | DT | <1 | 92.6 |
Traditional algorithm 2 | 45 | LPP | DSC | N/A | 85.1 |
Traditional algorithm 3 | 40 | RP | SRC | N/A | 81.6 |
Algorithm | 45 | Periodic extraction of calibration starting point | SVM | <1 | 98.9 |
DT, decision tree; LPP, partially preserving projection; RP, random projection; SRC, sparse classification; SVM, support vector machine; WARD, wearable sensor behaviour recognition database.
The choice of the classifier is significant for behaviour classification. Table 4 is the result of classification using different classifiers (SVM, DT, naive Bayes, K-nearest neighbour) after extracting the feature vector according to the algorithm of this paper. From Table 4, It can be seen that based on the feature vectors extracted by the algorithm in this paper, the SVM method can achieve a higher recognition rate.
Comparison of classification results of different classifiers
SVMs | 98.9 |
DT | 89.5 |
Naive Bayes | 86.9 |
K-neighbours | 93.9 |
DT, decision tree; SVM, support vector machine.
To further verify the rationality and scalability of the algorithm in this paper, this paper uses a motion capture system to record data under actual conditions. It applies the algorithm to the data set. Corresponding to the WARD data set, the database in this paper collects eight collectors (age 17–50 years old, four men, four women), eight daily behavioural actions (upstairs, downstairs, regular walking, clockwise walking, counterclockwise walking, jogging, turning left, turning right) actions, capture the data and extract the sensor data of the left wrist, right wrist, waist, left ankle and right ankle [11]. The data sample of the self-collected database is shown in Figure 7. Due to the data of the WARD data set, the sampling frequency is low (30 Hz), while the motion capture system used in this article has a high sampling frequency (90 Hz). It can be seen from Eq. (14) that if the resampling interval d of the fitting function is kept at 1, it will lead to the period data length being too long and the feature vector dimension being too large. To solve this problem, when the algorithm in this paper is applied to a self-collecting database, the sampling interval d in Eq. (13) is set to 3 (the sampling frequency of the database in this paper is three times that of WARD). Then we resample the fitting function and extract the feature vector. This adjustable resampling method reflects the applicability of the algorithm in this paper to the existing high-frequency motion capture system and reflects the advantages of functional data.
Analogous to the experimental approach of the WARD data set, we used the K-CV method to establish a training sample and test sample library (the self-collected data includes the data of eight experimenters [12]; therefore, the value of K here is 8, which is the 8-fold cross-validation method); and finally, we used the SVM for classification and recognition, and the recognition rate reached 98.75%.
Compared with the traditional discrete motion data processing method, this paper uses the FDA method to functionalise the discrete data to express the continuity characteristics of human motion data. It accurately calibrates the motion cycle according to the function of the motion data. Determining the starting point, and then approximately extracting the periodic sequence of the motion data based on the starting point, has certain advantages over the traditional window processing method. Thus, this article uses SVM and the periodic data sequence extracted by the algorithm discussed here to better classify and recognise the dynamic behaviour of the human body. The recognition rate compared with the traditional method is obtained. The algorithm's effectiveness in this paper has been verified on the public data set WARD and the self-collecting database.