Extraction of Knowledge from the Topographic Attentive Mapping Network and its Application in Skill Analysis of Table Tennis

In human motor skill research, movement skill is sometimes modelled as a hierarchical cerebellum model with feedback and feedforward functions that can adapt itself to environmental changes (Shiose et al., 2004). Kawato (1992) proposed the control model of Allen-Tsukahara as an internal model. When a difference exists between the desired trajectory and the realized trajectory of the movement, a difference signal is transmitted to Purkinje cells of the cerebellum and controls both movement output and starting timing. The Purkinje cells in the cerebellum control forward and inverse models for voluntary movement. Based on this view of the cerebellum’s contribution, in the present study, an internal model of the cerebellum is proposed in the form of a neural network acting through two kinds of processes, namely bottom-up processing of a signal flow to the integral representation of movement from a monofunctional layer, and top-down processing of the adjustment to the monofunctional layer from external observations.

The Topographic Attentive Mapping (TAM) network can be traced back to the ARTMAP family of models. The network structure consists of four layers: the feature layer, basis layer, category layer and class layer. The basis layer connects the feature layer to either the category layer or the class layer via bidirectional projections. A node in the basis layer network combines the bottom-input signal propagated from the feature layer via excitatory synapses with the top-down feedback signal controlled from the output layer via inhibitory synapses. Upon presentation of a training pattern, if the network produces inaccurate output, the attentional top-down signal modulates the synaptic weights in the class and basis layers via winner-take-all learning in order to minimize the difference between the network output and the desired output. At the same time, nodes can be added incrementally to the category layer until the classification accuracy on the training set reaches a satisfactory level. However, such an addition of nodes can lead to overlearning of the training data and hence, impair generalization performance. In order to address this problem, we propose a network pruning algorithm that can remove unnecessary nodes and links based on fuzzy information entropy (Hayashi and Williamson, 2001), and we apply an ensemble learning model to the TAM network to be able to make a recognition rate high by assigning selection weight to misclassified data. In the pruning algorithm of the TAM network, each input variable one-by-one is extracted and the fuzzy information entropy of each attribute is calculated to determine its importance. The importance of each attribute is computed indirectly as a reduction in the recognition rate when the corresponding link and node are removed. Besides improving recognition accuracy, the main advantage of using the pruning method within a TAM network is that it leads to a natural way of extracting knowledge from the network structure. The integration of the pruning mechanism in a TAM network also bears strong similarities with earlier work on learning in the context of fuzzy logic processes (Hayashi and Umano, 1993; Hayashi et al., 1996; Kosko, 1992) as our pruning approach incorporates the metaphysical concept of fuzzy ID3 (Hayashi, 1996; Hayashi et al., 1999; Umano et al., 1994) into the TAM network structure, implying that the data is mined in the format of a fuzzy rule. Ensemble learning with AdaBoost (Dietterich, 2000; Freund, 1995; Schapire, 2003) is adopted here to combine multiple TAM networks, each a weak classifier, and increase the recognition rate by differentially reweighting misclassified data. The final output is calculated with a majority rule after evaluating the multiple weak classifiers. According to this scheme, our algorithm may be considered as a form of a sensitive estimation method. Besides improving recognition accuracy, the main advantage of using the ensemble method within a TAM network is that it leads to a natural way of extracting a majority pattern from the data sets.

Input to our model takes the form of time-series extracted from motion picture data extracted from sensors (Kasai et al., 1994; Mochizuki et al., 2002). Perl and Baca (2003) employed the Kohonen Feature Map as a neural network for analysis of table tennis movement in order to characterize the strategic structure of table tennis. Parisi et al. (2015) proposed a learning system for providing feedback on a set of learned movements of powerlifting exercises captured with a Kinect device. The learning model was built upon a recursive extension of the Self-Organizing Map (SOM), MergeSOM. However, the proposed model did not aim to visualize a characteristic of the movement and produce a verbal description based on it. Therefore, it cannot be used to provide visualizations that would be useful to players. In our approach, we extract input attributes and skill rules of the forehand stroke of table tennis with the TAM network as an internal model (Hayashi et al., 2009; Maeda et al., 2014). The purpose of analyzing movement data with a TAM network is to visualize, for example, knowledge in the form of rules extracted from movement data. The aim of this research was to analyze the movement of players from observed data and help players to swing the racket based on acquired fuzzy rules. In order to do so, it is necessary to provide a verbal description of the observed data. A useful model of sports movements needs to provide for at least two kinds of capabilities: a high recognition rate and verbally-specified, easily understandable rules. Fuzzy rules based on languages can extract conjunctive variables with high membership values as important factors. This is a critical difference between fuzzy rules and methods based on statistical estimation. In this research, the technique rules are qualitatively specified as fuzzy rules from weights of network structure and necessary attributes are estimated to distinguish the skill level.

In the present study, the data of the two-dimensional (x, y) coordinates of all nine markers were analyzed with the TAM network. Since table tennis skills are better characterized with time-series, we created data sets by concatenating data across five consecutive frames. Another reason to use multiple frames as input was to be able to arbitrarily change the order presentation of each input variable to the TAM network. If the data had been input in the form of single frames, it would have not been possible to describe a series of racket swings. Therefore, past frames were also provided to the network in order to describe the series of swing movements. This is a method commonly used in Autoregressive Models (Pandit and Wu, 1983). In addition, our best recognition results were obtained when using series of 5 frames. The output of the model was the skill evaluation discretized to three levels. As a result, a data set consisted of 90 input variables (x and y coordinates of nine markers over five frames, that is 2 × 9 × 5) and three classes.

The training data (TRD) consisted of three kinds of players, i.e., two elite players who were selected from three elite players, two middle level players and two beginners who were selected from four beginners, and the testing data (CHD), which was constituted from the data of one elite player and one beginner. The results were strongly dependent on which kind of data was used during training and testing. Therefore, for beginner subjects, the correlation coefficient of the position coordinate was calculated at each marker and a data set D which included subjects with the highest correlation coefficient among four beginners in TRD and CHD was created, respectively. If data had been extracted randomly from all swing movements across all players, we would not have been able to define the racket motion because part of the movement might be missing. The number of frames in a swing from the start position to the end position for each player was different as the time for each swing movement was different for beginners, middle and elite players. Therefore, two players for the beginner and middle level and three for the elite level were used so that the amount of data for each level was approximately the same. Using the two extremes (elite and beginner) does not fully test the TAM. It is likely that these two categories could easily be separated by using simple rules. A more balanced and systematic approach would be to put aside one elite, one middle level and one beginner player for testing and the rest of athletes for training. All other participants would be then assigned to combinations to the testing group and the rest to training. This way all participants would become part of the testing group in all combinations with others, and they would be also part of the training group with others. Such a bootstrap method would provide an unbiased analysis.

The recognition rate obtained with the TAM network for TRD was 53.7%, and the recognition rate for CHD was 57.5%. This low accuracy could in part be due to a difference in the number of observations for each class. Therefore, for data set D, we constituted datasets by adding four consecutive frames to each initial input frame to get 5-frame long sequences and let the number of datasets increase by the added amount of data. The result is shown in Table 1. TAM (D) means the recognition rate of data set D, and TAM (D+) shows the recognition rate of the revised data set D with added data. Simultaneously, the recognition rates of C4.5, Native Bayes Tree (NBT), Random Forest (RF) are shown for comparing the TAM network with their data mining methods for data set D. In particular, C4.5 is an algorithm used to generate a decision tree developed by Quinlan (1993) and is an extension of the ID3 algorithm.

Table 1

Recognition Rates of Revised Data Sets:

The recognition rate for the TAM network using dataset D+ improved compared to when using D. The recognition rates of NBT and RF as to TRD equalled 100%, and that of C4.5 was 98.1%, which could be explained as overfitting given the corresponding low accuracy of testing data. The differences in recognition rates across training and testing sets were 67.2% and 74.6% for NBT and RF, respectively. Correspondingly, the difference between training and testing sets for C4.5 was 54.8%. This suggests again that NBT, RF and C4.5 overfitted the training data. On the other hand, differences in recognition rates between training and testing sets for the TAM(D+) and TAM(D) network were rather small, at the level of 18.4% and -3.8%, respectively. This suggests the TAM network did not overfit the training data. However, because the TAM network did not achieve high recognition rates on either the training or testing set, we re-trained the TAM network with Adaboost and showed the results in the next session.

Next, the sensitivity of markers was analyzed using a feature selection method. The selection of variables should be based on increases and decreases in the recognition rate. In general, the recognition rate decreases when reducing the number of variables. However, when unnecessary variables are removed this decrease is not significant. Conversely, when important variables are removed the recognition rate decreases greatly. Therefore, we should note the rate of increases and decreases in the recognition rate rather than its value. In the results of the experiment on variable selection, D+ was used as the data set. When all nine markers were used, the number of input variables was 90. That number decreased to 70 inputs when the number of the markers was decreased from nine to seven. In particular, we should pay attention to the variable of which the recognition rate decreased when reducing variables. The resulting recognition rates when removing markers are shown in Table 2. We showed that the average recognition rate was 10 times greater compared to the one obtained for TRD. When M1 and M2 were temporarily removed, the recognition rate of the TAM network decreased from 61.2% to 42.9% reaching the lowest value. Therefore, we concluded that the most important markers were M1 and M2. By the same procedure, inputs were sorted in importance as follows: M1, M2 → M7, M8, M9 → M5, M6 → M3, M4. The recognition rate was conversely increasing when M5, M6 and M3, M4 were removed. In addition, since pairs of markers attached to the same bones were located in the front side and back side of the bone, it was appropriate to delete pairs of markers at the same time. In fact, the number of markers could be reduced from 9 to 2 or 5, as pairs of markers showed similar movements.

Table 2

Sensitivity of Input Variables.

Lastly, fuzzy rules that characterized table tennis skills were extracted. The TAM network consists of four layers of hierarchical structure. The feature and basic levels represent a mono-functional mechanism, while the category and class levels represent a meta-concept. Using the structure of the TAM network, the relationship between the mono-functional skill and the meta-skill could be determined as a fuzzy rule.

Therefore, the J-th category node was first selected, where pjk became the maximum at each class node in the data set D+ under the condition of equation 21: (28)J={j|maxjpjk,k=1,2,3}.$$J = \{ j\,|\,\mathop {\max }\limits_j \,{p_{jk}},\,\,\,k = 1,\,2,\,3\} .$$

In addition, we calculated the real value wji of the J-th category node for each input as a singleton membership function: (29)wJi=∑h=1LwJihL,for∀i.$${w_{Ji}} = \frac{{\sum\nolimits_{h = 1}^L {{w_{Jih}}} }}{L},\,\,\,\,\,\,{\rm{for}}\,\,\,\,\,\forall i.$$

The set of linkages represented a fuzzy rule when we extracted linkages where wJi represented the maximum for each player. To express fuzzy rules for elite and beginner players, five categories with high pjk were extracted for each class. Since a category expresses a rule, these categories expressed the rules of the highest five pjk’s. In the first rule, wji of the marker on the racket in the horizontal direction was high, implying that this movement of the racket was important. The value of wji of each marker in the vertical direction was low, meaning that the swing movement was stable and without rough wavy movements. In both horizontal and vertical directions after the third rule, the value of wji was constant and the swing was stable. On the other hand, the vertical rise and fall motion was a salient feature of beginners’ motion patterns and was extracted as the first rule. The horizontal movement at the shoulder and the elbow was remarkable in the second rule, implying “a movement to delay the body”. Figure 4 shows the first rule of the elite player and beginner. In this Figure, the difference between the elite player and the beginner is shown very conspicuously, making it easy to formulate table tennis skills as a rule.

The recognition rate achieved with the TAM network was better than that obtained with other data mining methods. However, the recognition rate was still fairly low. Therefore, we applied the Adaboost algorithm, which is a kind of an ensemble learning model, to the TAM network in order to improve its recognition rate. First, a dataset was built so that the number of data samples of each dataset became approximately the same across classes. By adjusting the dataset in this way, the number of elite players in TRD of the data set D++ became 78, the number of middle level players became 73 and the number of beginners became 98. On the other hand, as to CHD of data set D++, the number of elite players and beginners was 54 and 40, respectively. The recognition rate of TRD was 61.2%, while the recognition rate of CHD was 43.0%. The recognition rate was computed as the average across 10 repetitions of the experiment. For dataset D++, three kinds of data groups were constructed as Adaboost was by default defined for two-class problems. That is, dataset D++ was partitioned into three groups with two classes, i.e., D1 which included the beginners and the others, D2 that comprised the middle level players and the others, and finally D3 composed of the elite players and the others. These data sets were analyzed by the Adaboost type TAM network with epoch = 3, α = 0.0000001, λ = 0.33.

The results are summarized in Table 3. The average recognition rate was 10 times of the data sets. As to TRD of dataset D1, the Adaboost algorithm was repeated three times, and 149 data were selected as misclassified TRD at the first step of the algorithm, and 42 data were selected as misclassified TRD at the second step. In the same way as to the data set D2, 149 data were selected as misclassified TRD in the first step of the algorithm, and 44 data were selected as misclassified TRD in the second step. As to the data set D3, 149 data were selected as misclassified TRD in the first step, and 42 data were selected as misclassified TRD in the second step. The average recognition rate of the Adaboost-type TAM network for TRD improved to 73.1%, which was better than 69.3% of the original TAM network. As to CHD, the recognition rate of Adaboost type TAM network increased to 65.5% as the recognition rate of the original TAM network was 58.2%. A t-test applied to these results showed that Adaboost-type training of the TAM network improved accuracy over the original TAM network (p = 0.014).

Table 3

Recognition Rate by the Adaboost Type TAM Network

In this section, we will discuss the evaluations of the model. First, the following observations with regard to the coordinates of positions can be made from Figures 1 and 2:

Since the correlation coefficients of the horizontal direction (x) of the markers for elite and middle level players were higher than 0.9, it can be inferred that the swing movements in the horizontal direction of each elite and middle level player were very similar. At the same time, the correlation coefficient for the vertical direction (y) was higher than 0.6, and thus, the movement in the vertical direction was also similar. However, the correlation coefficients for beginners were less than 0.1 in both horizontal (x) and vertical (y) directions, and consequently, the movements of the beginners were different from each other. The recognition accuracy of NBT, RF and C4.5 on the training set was very high (Table 1), which can be explained as overfitting given the corresponding low accuracy on the test data. Since the beginner category is hard to clearly define, it is difficult to obtain high recognition accuracy using a discriminatively trained model. Differences in recognition rates across training and test sets were very large for NBT, RF and C4.5. This suggests again that NBT, RF and C4.5 overfitted the training data. On the other hand, differences in recognition rates between training and testing sets for the TAM(D+) and TAM(D) network were rather small. This suggests that the TAM network did not overfit the training data.

Next, the following observations with regard to the racket swing motion can be made from Figures 1 and 2:

Using a feature selection method, the sensitivity of markers was analyzed. We concluded that the most important markers were Ml and M2, and inputs were sorted in importance as follows: Ml, M2 → M7, M8, M9 → M5, M6 → M3, M4. Based on these results, it could be stated that important variables to evaluate player’s ability are firstly, 1) the acromioclavicular joint and 2) the acromion, and secondly, markers 7 to 9 located on the racket. These results are consistent with the conclusions reached from Figures 1 and 2.

Lastly, the following observations with regard to the swing speed can be made from

Figures 1 and 2:

Figure 4 shows the first rule obtained for both an elite player and a beginner. In the first rule of the elite players, the weight value of the racket in the horizontal direction was high, implying that this movement of the racket was important. The weight value of each marker in the vertical direction was low, meaning that the swing movement was stable and without rough wavy movements:

On the other hand, the vertical rise and fall motion was a salient feature of beginners’ motion patterns and thus, it was extracted as the first rule.

From the TAM network, regarding the width of the swing pattern in the horizontal direction (x), elite players appear to swing rather compactly, whereas beginners make large swings. In addition, the difference between the elite player and the beginner is shown very conspicuously, making it easy to formulate table tennis skills as a rule. In particular, when a player hit a ball with the racket while forming a straight line with his head, shoulder and foot at the center of his body to create a stable axis of rotation, the shoulder of the beginner did not remain stable. The swings of the elbow, wrist and racket were also wide in the vertical direction. The horizontal movement at the shoulder and the elbow was remarkable in the second rule, implying either “a movement to meet a ball with the racket” or “a movement to delay the body”.

Fuzzy rules based on languages can extract conjunctive variables with high membership values as important factors. As a result, with the TAM network, input attributes and technique rules were extracted in order to classify the skill level of players of table tennis from the sensor data. However, we should explore the structure of the resulting internal model in order to better understand how to improve table tennis skills in the future.

In this paper, a dataset of forehand strokes of table tennis was analyzed with a TAM network and a boosted TAM network. In addition, fuzzy rules describing the skill of forehand strokes were extracted depending on the player’s performance level. Future work should include exploring the structure of the resulting internal model in order to better understand how to improve table tennis skills.