Application of data mining in basketball statistics

For the linear regression, if a linear relationship exists between independent variables and dependent variables, it will meet the following equation: (4) (xi)=θ0+θ1xi1+…+θ1xij+θ1xin \left({{x_i}} \right) = {\theta _0} + {\theta _1}{x_{i1}} + \ldots + {\theta _1}{x_{ij}} + {\theta _1}{x_{in}}

There are m samples, and each sample has n dimensions, where x_i is the ith input sample vector, h_θ (x_i) represents the output value, x_ij represents the jth component of the ith input sample vector and θ_j represents the jth component of the θ vector. The prediction objective is to minimise the loss function of the model: (5) J(θ)=12∑i=1m(hθ(xi)−yi)2 J\left(\theta \right) = {1 \over 2}\sum\limits_{i = 1}^m {({h_\theta}\left({{x_i}} \right) - {y_i})^2}

The least square method is used to estimate the parameters. This method can deduce the optimised parameters after the model training, so as to predict outcomes by regression.

XGBoost Algorithm:

XGBoost is a member of the boosting family of ensemble learning, which is an improvement of the boosting algorithm based on the Gradient-boosted Decision Tree (GBDT). GBDT fits to the residual by using the negative gradient of the model on the data as the approximate value of the residual. XGBoost attempts to fit to the residual of data. However, this algorithm uses the second-order Taylor expansion to fit loss residuals of model, and at the same time improves the loss function of the model and adds a regular term of model complexity [10].

XGBoost has improved performance over GBDT, and its performance can also be seen in various competitions. Under a reasonable parameter setting, GBDT can only achieve satisfactory accuracy after generating a certain number of trees. In some way, when the dataset is complex, the model may be iterated for thousands of times. However, XGBoost can solve this problem better by using parallel CPUs. This is the mathematical principle of XGBoost.

Boosting is a method to transform a weak classifier into a strong classifier, whose function model is superposed. To be specific: (6) D={(xi,yi)}(|D|=n,xi∈ℝ) D = \left\{{\left({{x_i},{y_i}} \right)} \right\}\left({\left| D \right| = n,{x_i} \in {{\rm{\mathbb R}}}} \right) (7) y^i=ϕ(xi)=∑k=1Kfk(xi), fk∈F {\hat y_i} = \phi \left({{x_i}} \right) = \sum\limits_{k = 1}^K {f_k}\left({{x_i}} \right),\quad {f_k} \in F where D represents a dataset, m dimensions and quantity is n. Specifically, each tree is a tree model, which can be expressed as follows: F={f(x)=wq(x)}(q:ℝn→T, ω∈ℝT) F = \{f\left(x \right) = {w_{q\left(x \right)}}\} (q:{{{\rm{\mathbb R}}}^n} \to T,\;\omega \in {{{\rm{\mathbb R}}}^T}) where q(x) represents the mapping relationship between the sample x and the leaf nodes of the tree model, and ω is the predicted value used to fit to the samples belonging to the respective leaf nodes in the tree model. From the second round, for each training, the residual between the predicated and actual values of the last round is input. The result of the last leaf node is also the prediction of the residual. The sum of residuals in all rounds is what we want.

The neural network algorithms are widely used in all subdomains of artificial intelligence. They are briefly introduced in the literature, including the explained version. w_ij is considered as the strength of connection between neurons i and j. Basically, a neural network includes three groups, namely visible set V, hidden set Hand neuron output set O. The set V consists of neurons that receive signals and pass them to hidden neurons in the set H. In terms of the mathematical principle for deep neural network learning, ReLU function is used as the main activation function in this paper [11, 12, 13]: (11) f(yi)={yi,yi>00,y≤0 f\left({{y_i}} \right) = \left({\matrix{{{y_i},} & {{y_i} > 0} \cr {0,} & {y \le 0} \cr}} \right.

Data were pre-processed, including filling the missing value, making the variable name uniform and carrying out variable standardisation. The mean value was used to fill in the missing data. The data from different sources were uniformly processed and the player data were standardised for min − max to make them to fall into the range of [0,1]. As this model selected a fixed number of historical games, each statistic has a comparable computational cost across seasons. The weights for each game are shown in Figure 1. Since the first 10 games are ignored in each season, the number of historical games is now 75,239.

Except for the variables directly used to calculate FPTS, other variables were selected for models to quantify the player ability by using more advanced statistical methods. The details are discussed in part 2.

Using data from recent games allows a more objective and accurate prediction of the results of players’ abilities. Therefore, the weighted mean of the past 10 games was used to obtain each variable. The relevant theory shows that the mean weight of games increases linearly with the number of games. In this paper, the square root and linear and square modes were used for quantitative evaluation. It is necessary to normalise the weights such that the sum of the weights is 1. As shown in Figure 1, the weighted square mode is a better weighting method. Accordingly, it was considered as the best option in this paper.

For the consistency of feature variables, the standard deviation of the FPTS variable over the last 10 games was defined as FPTS_std, while salary information from DraftKings was also defined as the Salary variable. Before a game, the participation of a player in the game cannot be determined from the model. Thus, it is necessary to calculate the value of the model's feature variable according to the published player sheet [14] before the game.

Fig. 2

Because of the different predictive abilities of features on game outcomes and correlations between features, the features should be filtered and ranked. Taking the field goal (FG) as an example, it is highly correlated with effective field-goal percentage (eFG%) because the latter considered far fewer free throw points. Furthermore, some variables have multiple collinearity, such as three pointers (3P), three pointers (3PA) and three-point percentage (3P%). In this paper, the Pearson correlation coefficient was used for screening features. With the setting correlation threshold of 0.90, the following six features were screened out: three-point shot (3PA), defensive rebound (DRB), field goals attempted (FAG), field-goal percentage (FG%) and offensive rating (ORtg). In addition, variables without predictive ability are directly ignored in models. The gradient descent method was used in models to evaluate and quantify the feature significance of 34 variables. Using feature ranking, such features as dummy variables of position (SG, F, C), three pointers (3P) and three doubles (TD) were excepted. Finally, the remaining 29 variables were used as selected features for regression, gradient enhancement and deep learning.

Fig. 3

The results of linear regression for all variables, using three different datasets, are shown in Table 1. A 5-fold cross validation was implemented when the linear regression model was trained. According to the linear model regression prediction, the minimum value of RMSE is 9.5356.

For the XGBoost model, the parameters should be adjusted and optimised during model training. First, the RMSE produced by XGBoostRegressor using the 29 variables selected above for any parameter adjustment is 9.0316. Then the Bayesian optimisation algorithm was used to adjust the parameters and improve the model. For the XGBoost's parameters to be optimised, the distribution of the variables was set in a uniform form, except for the learning rate, which distributes in a logarithmic uniform form. Parameters were randomly selected from the first five parameters and iteratively improved from the optimal performance parameters’ set. During each iteration, the 5-fold cross validation was applied to evaluate a given parameters’ set. Through a series of iterations, the resulting optimal performance parameters’ set was:

Max_depth: 5, n_estimators: 354, n_child_weight: 0, gamma: 0.8, learning_rate: 0.0152. The use of these parametric adjustment models will result in better performance (RMSE is 8.9581).

In terms of the neural network model, Figure 4 shows the learning process of the model. A total of 20% of the training data was retained as validation data. It can be seen that the model soon starts to overfit, and verification losses are different from training losses. To prevent the model from overfitting, a loss layer was added, which randomly ignored 40% of the data points before feeding them forward to the last layer. In this paper, the EarlyStopping method in the Keras.callbacks model was applied. If verification is lost for no more than five periods, learning is terminated. This solution improved the model performance, with the RMSE reduced from 9.0678 in the original model to 9.0387.

Fig. 4

Finally, the performance of these three models was compared, as shown in Table 2, where the models in the first and third rows calculated the mean value by using the defined linear combination of coefficients. In the fourth and fifth lines, the XGBoost and neural network algorithms used the selected 29 variables and quadratic weights. Obviously, the XGBoost algorithm shows the best predictive performance, with the lowest RMSE (8.9581) and MAE values (6.8486). According to the training results of these three models, the XGBoost regression model with hyperparameter adjustment was selected as the final prediction model for basketball games.

After the prediction calculation by models, and 5-fold cross validation, the consistent RMSEs (8.9910, 9.0522, 9.0148, 8.9351, and 9.0831) were obtained. In order to further improve the robustness of the model, the effect of small changes in input data on the performance of the model was also studied. First, the Gaussian noise was created by using mean 0 and variance 1, which was then scaled to the range of [0, 0.2] and added to the continuous variable in the original input. When the 5-fold cross validation was performed on the independent variable evaluated by using the noise, the losses seem to have barely budged, with RMSE and MAE only increasing to 6.8964 and 9.0153, respectively. As a result, the model was proven to have good robustness. The final model showed that the RMSE increased by nearly 10%, from 9.9434 in the linear model to 8.9581.

For the model, t-test equation was used to verify that the RMSE of the XGBoost model was better than the RMSE (9.9434) of the baseline model, and the former has a simple mean value. As the null hypothesis is set, the XGBoost predictor's performance is equal to the average performance of baseline model of the sample. Assuming that the two groups follow a normal distribution, the RMSE can be calculated based on the standard deviation σ and sample size n after the 5-fold cross validation according to the definition, because the central limit theorem can be applied to the quantity involving the average value. In this case, the original result is: 8.9910, 9.0522, 9.0148, 8.9351, 9.0831 8.9910,{\kern 1pt} 9.0522,{\kern 1pt} 9.0148,{\kern 1pt} 8.9351,{\kern 1pt} 9.0831

Assuming that L˜ \tilde L is the RMSE of XGBoost's predicted value and L₀ is the RMSE of the baseline model, the t statistic is: t=L˜−L0σn=9.9434−8.95810.04055=121.51>15.54 t = {{\tilde L - {L_0}} \over {{\sigma \over {\sqrt n}}}} = {{9.9434 - 8.9581} \over {{{0.0405} \over {\sqrt 5}}}} = 121.51 > 15.54

The statistical significance was 0.1%, with four degrees of freedom and a critical value of 15.54. Therefore, the weighted mean, feature engineering and XGBoost regression can be used to improve the accuracy of predicted outcomes of NBA games.