Early Warning System for Debt Group Migration: The Case of One Commercial Bank in Vietnam

Credit quality assessment is an important aspect in the core business of a bank, as it allows banks to assess the repayment ability of borrowers and determine their loan repayment capacity. Credit scores are a digital representation of a borrower’s credit history and help lenders assess the risk level of borrowers. There are different types of credit scores, each providing specific information to bank managers about the borrower’s credit history and behavior. This study will discuss three typical types of credit scores: Application Score (A Score – credit score for new customers), Behavioral Score (B Score – credit score regarding the repayment behavior of existing customers), and Collection Score (C Score – credit score regarding the debt recovery ability of defaulted customers).

The first type of credit score is the A Score, which is based on information provided by the borrower on the credit application. This score considers key factors such as the borrower’s income, employment history, credit history, and debt-to-income ratio. Banks use this score to assess the borrower’s repayment ability and decide whether to approve the credit application. The A Score is crucial in evaluating the borrower’s initial debt repayment capability.

The second type of credit score is the B Score, which reflects the repayment behavior of customers and is also known as the performance credit score. This score is based on the borrower’s credit behavior during the credit relationship with the bank, such as payment history, credit utilization, length of credit usage, and types of credit used. The B Score provides banks with information about the borrower’s credit habits and their ability to repay debts on time. This score is essential in assessing the borrower’s credit risk over an extended period.

The third type of credit score is the C Score, which is used to assess the debt recovery ability of borrowers and is typically applied to customers showing signs of default.

The C Score is based on monitoring overdue or non-payment debts and the collection activities related to resolving these debts. It allows the bank to evaluate the borrower’s ability to handle overdue debts and bad debts and decide whether to extend credit to the borrower. The collection credit score is an important indicator to assess the borrower’s ability to manage and resolve debts as well as their ability to recover to a good credit status.

All three types of credit scores can reflect the lifecycle of a customer’s credit relationship with the bank. According to Figure 1, a new customer submitting a credit application will be scored based on the A Score. If approved, the customer is granted credit and begins the repayment process with the bank. This process is monitored by the B Score. If the customer defaults, the bank will initiate collection efforts and assess the debt recovery ability through the C Score.

Figure 1.

The lifecycle of a customer’s credit relationship with a bank. (Source: Author’s own research)

Within the scope of this study, an early warning system for the shift in the debt group has a close relationship with the B Score and C Score. The early warning system needs to provide signals indicating existing customers at risk of default in the future based on observable credit behaviors at present, thereby enabling the bank to proactively deal with potential losses and implement provisions for bad debts. Furthermore, for customers who have defaulted or have overdue debts, the system also helps identify customers from whom the bank can recover debts and restore the good debt group, thus providing timely support measures for customers.

According to Circular 11, customer loans are classified into risk categories as follows:

The debt groups from 1 to 5 represent a decreasing order of credit quality, and based on the factors of overdue days, there can be a phenomenon of debt groups potentially shifting to better or worse groups. For example, if a customer fails to repay the debt within 3 months (> 90 days), it will be classified as Group 3 debt. However, if the customer makes full payment of principal and interest for one month, the remaining overdue debt will be below 90 days and will be classified as Group 2 debt.

Based on these foundations, we proposed the following criteria to develop an early warning system:

Regarding credit quality improvement, we have not found specific criteria for precise evaluation. However, most banks will recognize customer credit quality improvement based on the following two factors:

Therefore, if a customer meets both of the above criteria within a total period of 6–12 months, the bank will recognize the improvement in the customer’s credit quality.

To achieve the objective of forecasting or warning with the smallest possible error, early warning models for credit risk need to be built on solid statistical and economic theories. Various studies have proposed multiple methods for constructing high-reliability early warning models. However, there are two most common methods being applied in many countries: the signal method and the quantitative method.

2.4

Machine learning models

2.4.1

Logistics Regression (LG)

This is the most fundamental model. To gain a better understanding of logistic regression, we can start with a linear regression model in the following form:

y=wTx $${\rm{y}} = {{\rm{w}}^{\rm{T}}}{\rm{x}}$$

where y represents the dependent variable, also known as the target variable; x represents the independent variables; and w^T is a matrix of weights corresponding to each independent variable, also known as regression coefficients. Linear regression models are typically used when y is a continuous variable with a defined range (−∞; + ∞).

However, in the context of this study, the target variable is a binary variable with two values, 0 and 1. If we still apply the linear regression model (Gujarati and Porter, 2009) in this case, the predicted outcome y (denoted as y′) will fall into two scenarios:

If ŷ ∈ (0; 1), it can be interpreted as the probability of y = 1. This idea corresponds to the Linear Probability Model (LPM).

If ŷ < 0 or ŷ > 1, it cannot be interpreted as a probability. If we constrain the values of ŷ to the range (0; 1), it would result in losing the interpretation of the slope coefficients in the linear regression model.

To address this issue, the component w^Tx will be passed through an activation function to transform its value range. At this point, the model can be rewritten as follows:

z=fwTx $${\rm{z}} = {\rm{f}}\left( {{{\rm{w}}^{\rm{T}}}{\rm{x}}} \right)$$

The notation z replaces y because y is a binary variable with only two values, while z is a variable with a different value range depending on the type of activation function.

If the activation function transforms w^T x from the value range (-∞; +∞) to (0;1), then the model above becomes the logistic regression model. The commonly used activation function is the Sigmoid function (specifically, the Logit function), and the logistic regression model is written as follows:

z=11+e−wTx $${\rm{z}} = {1 \over {1 + {{\rm{e}}^{ - {{\rm{w}}^{\rm{T}}}{\rm{x}}}}}}$$

When the predicted outcome from the model, ẑ, falls within the range (0; 1), it represents a probability. The probability threshold is typically chosen as 0.5, so if ẑ ≥ 0.5, then ŷ = 1, and if ẑ < 0.5, then ŷ = 0.

2.4.2

Support Vector Machine (SVM)

The SVM model is constructed by creating a plane (or hyperplane) to separate the data in a way that optimizes the distance between data points and the separating plane. In general, the distance of a point with coordinates x₀ to the hyperplane with the equation w^Tx + b = 0 is determined as follows:

wTx0+bw2 $${{\left| {{{\rm{w}}^{\rm{T}}}{{\rm{x}}_0} + {\rm{b}}} \right|} \over {{{\rm{w}}_2}}}$$

The objective of SVM is to find the values of w and b such that the distance from the separating plane to the nearest data point is maximized. Therefore:

w,b=argmaxminynwTxn+bw2==argmax1w2minynwTxn+b $$\matrix{ {\left( {{\rm{w}},{\rm{b}}} \right) = \arg \max \left\{ {\min {{{{\rm{y}}_{\rm{n}}}\left( {{{\rm{w}}^{\rm{T}}}{{\rm{x}}_{\rm{n}}} + {\rm{b}}} \right)} \over {{{\rm{w}}_2}}}} \right\} = } \hfill \cr { = \arg \max \left\{ {{1 \over {{{\rm{w}}_2}}}\min {{\rm{y}}_{\rm{n}}}\left( {{{\rm{w}}^{\rm{T}}}{{\rm{x}}_{\rm{n}}} + {\rm{b}}} \right)} \right\}} \hfill \cr } $$

The optimization problem can be simplified as follows (Bishop, 2006):

w,b=argmin12w22subject to:1−ynwTxn+b≤0,∀n=1,2,...,N $$\matrix{ {\left( {{\rm{w}},{\rm{b}}} \right) = \arg \min {1 \over 2}{\rm{w}}_2^2} \hfill \cr {{\rm{subject\;to:}}1 - {{\rm{y}}_{\rm{n}}}\left( {{{\rm{w}}^{\rm{T}}}{{\rm{x}}_{\rm{n}}} + {\rm{b}}} \right) \le 0,\forall {\rm{n}} = 1,\,2,...,{\rm{N}}} \hfill \cr } $$

where y_n = 1 or y_n = −1 depending on the binary label of data point n.

In practice, data are often difficult to perfectly separate linearly. As a result, there will be the occurrence of noisy data points. The optimization problem of SVM is then rewritten as follows:

w,b=argmin12w22+C∑n=1Nεnsubject to:1−εn−ynwTxn+b≤0,∀n=1,2,...,N−εn≤0,∀n=1,2,...,N $$\matrix{ {\left( {{\rm{w}},{\rm{b}}} \right) = \arg \min {1 \over 2}{\rm{w}}_2^2 + {\rm{C}}\mathop \sum \limits_{{\rm{n}} = 1}^{\rm{N}} {{\rm{\varepsilon }}_{\rm{n}}}} \hfill \cr {{\rm{subject\;to:}}1 - {{\rm{\varepsilon }}_{\rm{n}}} - {{\rm{y}}_{\rm{n}}}\left( {{{\rm{w}}^{\rm{T}}}{{\rm{x}}_{\rm{n}}} + {\rm{b}}} \right) \le 0,\forall {\rm{n}} = 1,\,2,...,{\rm{N}}} \hfill \cr { - {{\rm{\varepsilon }}_{\rm{n}}} \le 0,\forall {\rm{n}} = 1,\,2,...,{\rm{N}}} \hfill \cr } $$

where C is the weight that determines the trade-off between optimizing the distance and handling noise. The ε_n of a data point reflects the level of noise:

If the data point is far from the separating plane and correctly classified, then ε_n = 0.

If the data point is close to the separating plane but still correctly classified, then 0 < ε_n < 1.

If the data point is misclassified, then ε_n> 1.

Another issue is that the data may not be linearly separable. However, it is possible to use certain activation functions to map the data to a different feature space where it can be linearly separable. These activation functions are called kernel functions. Common kernel functions are defined as follows:

Linear kernel: k(x, z) = x^Tz

Polynomial kernel: k(x, z) = (r + γx^Tz)^d

Radial Basic Function (RBF) kernel: kx,z=e−γx−z22,γ>0 ${\rm{k}}\left( {{\rm{x}},{\rm{z}}} \right) = {{\rm{e}}^{ - {\rm{\gamma x}} - {\rm{z}}_2^2}},{\rm{\gamma }} > 0$

•

Sigmoid (tanh) kernel: k(x,z)=tanh (γx^Tz + r)

For these kernel functions, r and γ are parameters that can be tuned. In the case of the Polynomial kernel, an additional parameter d represents the degree of the polynomial function (d is a natural number). It is worth noting that the Linear kernel does not have any adjustable parameters.

2.4.3

Decision Tree (DT)

Decision tree is a classification model that uses variables to divide data into smaller samples. These samples are further divided in a similar manner until the labels of the data (target variable) can be predicted. Some related concepts in DT, as shown in Figure 2, include:

Root node: It is the first node in the DT, where the data are not yet divided.

Internal node: These are intermediate nodes in the DT. They are the result of the division at the previous node and serve as data for further division by a specific variable.

Leaf node: It is the final node that is no longer divided by any variable. At the leaf node, the DT needs to make a prediction for the label of the data.

Depth: Each division in the DT forms a depth in the tree, indicating the level of divisions.

Figure 2.

A decision tree classifier (Source: Author’s illustration)

At each internal node, the DT decides whether to continue dividing the data and if so, which variable to choose. Generally, the division process stops if the node contains data points with the same label (pure node). However, this may result in a very deep tree or the inability to find a variable for division. Therefore, limiting the depth and the number of leaf nodes in the DT is crucial.

The next issue is determining which variable to use for division. In DT, there are two basic criteria used to evaluate variables: Entropy and Gini (Rokach and Maimon, 2008; Shalev-Shwartz and Ben-David, 2014).

Entropy at any given node is calculated as follows: Entropy=−∑ipi×log2pi $${\rm{Entropy}} = - \mathop \sum \limits_{\rm{i}} {{\rm{p}}_{\rm{i}}} \times \log 2\left( {{{\rm{p}}_{\rm{i}}}} \right)$$

Entropy has a minimum value of 0, indicating a pure node where all data points have the same label. The larger the entropy, the more diverse the distribution of labels in the node.

Gini at any given node is calculated as follows: Gini=1−∑ipi2 $${\rm{Gini}} = 1 - \mathop \sum \limits_{\rm{i}} {{\rm{p}}_{\rm{i}}}^2$$

Gini also has a minimum value of 0. Similar to entropy, when Gini = 0, it indicates a pure node, and as Gini increases, the distribution of labels in the node becomes more diverse.

To select the variable for division, the Entropy (or Gini) is computed for the initial node before the division. Then, the Entropy (or Gini) is calculated for each node after the division. The information gain of a variable is determined as follows:

IG=EInitial−∑nwnEn $${\rm{IG}} = {{\rm{E}}_{{\rm{Initial}}}} - \mathop \sum \limits_{\rm{n}} {{\rm{w}}_{\rm{n}}}{{\rm{E}}_{\rm{n}}}$$

where E_Initial is the Entropy (or Gini) at the initial node, E_n is the Entropy (or Gini) of node n after the division, and w_n is the weight of the number of observations in node n relative to the number of observations at the initial node. Each variable will have a different information gain, and the variable with the highest information gain is chosen for the division.

2.4.4

Random Forest (RF)

Random forest is a model that consists of multiple decision tree (DT). On a dataset, RF performs random sampling with replacement to create n data samples, each with a smaller or equal size compared to the original sample. For each obtained data sample, a DT model is built and used to predict the entire original dataset. RF includes n constructed DTs and utilizes a voting mechanism to determine the label for each data point. Thus, the essence of RF lies in multiple DTs, where RF has the additional task of labeling through each DT and aggregating the results to obtain the final label assignment.

Due to the execution of multiple DTs, the number of DTs should be carefully considered when implementing RF to ensure computational efficiency, especially for large datasets. This is an important parameter of RF. Other tuning parameters for RF are similar to those of DT, such as the maximum number of leaf nodes and the depth of the tree. RF also has adjustments regarding the sample selection for conducting the DTs.

2.4.5

Parameters tuning

In general, each model has its own advantages to achieve the goal of classifying data. Fine-tuning the models is necessary to maximize the benefits of each model. Table 2 summarizes the important parameters that need to be adjusted to ensure the optimal performance of the model.

Table 2.

Summary of parameters of models (Source: Author’s compilation)

Model	Parameter	Description
LG	None	The baseline model is a linear regression model combined with the sigmoid (logit) activation function, so no tuning is required
SVM	Kernel function	The activation function used to transform data into a different feature space for linear separation includes Linear, Polynomial, Sigmoid, and RBF
	C	The coefficient for balancing the weight between distance and noise
	d	The degree parameter when using the Polynomial kernel, which takes a natural number value
	γ	The gamma parameter for Polynomial, Sigmoid, and RBF kernels, which takes a non-negative value
	r	The intercept for the Polynomial and Sigmoid kernels
DT	Depth	It is necessary to limit the depth of the DT to avoid overfitting and reduce computational cost
	Number of leaf nodes	It is necessary to limit the number of leaf nodes of the DT to avoid overfitting and reduce computational cost
RF	Depth	It is necessary to limit the depth of each DT to avoid overfitting and reduce computational cost
	Number of leaf nodes	It is necessary to limit the number of leaf nodes of each DT to avoid overfitting and reduce computational cost
	Number of DTs	The number of DTs in Random Forest Classifier (RF) needs to be considered for computational cost when the number is too high

During the data preprocessing phase, features are created to enhance the model’s interpretability. We employ three transformation techniques as follows:

With the transformed variables, we proceed to perform binning for each variable and calculate the Weight of Evidence (WoE) and Information Value (IV) values (Greiff, 1999; Siddiqi, 2006) as follows:

where i denotes the bin, k represents the total number of bins, n_i indicates the number of observations with target variable equal to 0 in bin i, p_i represents the number of observations with target variable equal to 1, and N_i represents the total number of observations in bin i.

In addition, the GINI criterion is also determined based on the area under the receiver operating characteristics (AUROC) using the following formula:

GINI = 2 × AUROC − 1

where AUROC is the area under the ROC curve illustrated in Figure 5.

Figure 5.

ROC and AUROC (Source: Author’s illustration)

The author conducted a Delphi survey with over 300 DX experts in SMEs in Vietnam using the Foresight approach to assess the features, challenges, and business vision of DX. The study also forecasts the future demand for DX in SMEs and the most important activities for implementing it. To calculate the sample size and determine the reliability level, the author used Yamane’s (1967) sampling technique as a reference. The findings of the analysis were presented in detail in the study “Building a Digital Transformation Strategy for SMEs in Vietnam – Approaching Foresight” (Tuan, et al., 2023).

Most businesses need government assistance and tools during the DX process. In terms of support tools, most businesses have not used a comprehensive support system and desire one with the following capabilities.

The result of each bin corresponds to a WoE value. Assigning this WoE value to the corresponding bins in the original dataset will yield a series of WoE variables. These WoE variables also play a significant role in predicting the target variable of the models.

IV and GINI are used to measure the predictive power of features. IV and GINI are directly proportional to predictive ability. Since there are many additional variables created, including all of them in the model can increase computational cost. Therefore, it is necessary to filter out variables with a good predictive power. We perform variable filtering based on the IV criterion (Siddiqi, 2006) with the following thresholds:

Therefore, this study will choose the IV threshold of ≥0.02 to limit the number of variables used in model estimation.

The data used for model estimation are historical data divided into a training set and an evaluation set. The data splitting is performed using stratified sampling based on the target variable labels, with a ratio of 70% for training and 30% for evaluation. The models are estimated using the Python libraries of Scikit-learn. The model tuning parameters are described in Table 3.

Another important parameter that needs to be declared in all models is class_weight = ‘balanced’. This is an important parameter to give higher weight to the target variable = 1 labels that are correctly predicted, as there is an imbalance in the data.

Model tuning will be based on optimizing the accuracy of predictions, specifically, creating a confusion matrix and calculating relevant metrics. According to Fawcett (2006), a confusion matrix reflects the true and false similarities between the predicted results and the actual data. Therefore, a confusion matrix is a 2×2 matrix with columns representing the number of model-predicted observations and rows representing the number of actual observations, as follows in Table 4.

It is important to note that both FP and FN reflect the model’s errors, but the significance of these errors depends on the specific nature of what the model is predicting. In the specific case of this research, the emphasis on importance will focus on the following two errors:

The matrix can be transformed into a ratio (probability) form by dividing each element by the sample size (P + N) to obtain the following ratios: TPR, FNR, FPR, and TNR, in which R stands for Ratio. In addition, there are several criteria computed from the confusion matrix that are valuable for evaluating the model:

where:

β = 1: No weight is assigned, similar to F₁.

β > 1: Recall is more important than Precision.

β < 1: Precision is more important than Recall.

Powers (2011) pointed out that F₁ score is not a good measure, especially when dealing with imbalanced data, as it disregards True Negatives (TN) results. Hand and Christen (2017) also criticized the overemphasis on Precision and Recall when using F1 score. In reality, different types of misclassifications have different consequences. Chicco, et al. (2021) proposed using the Matthews Correlation Coefficient (MCC) to evaluate the model, with the formula:

Therefore, to ensure effective model adjustment, the study uses the MCC criterion. However, due to the nature of credit risk, the adjustment and selection of the optimal model in operations should also consider the following criteria:

At this phase, a comprehensive evaluation of the model will be conducted, considering both evaluation criteria: MCC and criteria relevant to credit risk management. Specifically, for the B Score customer group, the adjustment will be based on F-Recall, while for the C Score customer group, the adjustment will be based on F-Precision. Additionally, the comparison, analysis, and selection process should consider the relationship between these criteria across the datasets. The final selected model must ensure stability in both the training and test sets, while optimizing the necessary magnitude of the important criteria

During the implementation of an early warning system, besides the reported warning signs, the actual shift in the customer’s debt group needs to be reviewed to determine if it aligns accurately with the early warnings. This allows for an evaluation of the system’s effectiveness in issuing timely warnings. This study also performs simulations to assess the operational capability of the early warning system on a completely new dataset compared to the datasets used to build the model.

The cases of each model during the tuning process will be sorted in ascending order according to the relevant criteria on the training set, and then, the performance trend of the model on the evaluation set will be assessed. The results are as follows Figure 6.

Figure 6.

Logistic regression results (Source: Author’s own calculation)

According to Figure 7, it can be observed that the LR, when used for both B Score and C Score datasets, does not exhibit significant overfitting as the MCC criterion does not show a significant difference between the training and evaluation sets. However, the effectiveness of the model remains relatively modest. Similarly, there is no difference in the F-Recall criterion for the B Score and F-Precision criterion for the C Score between the two datasets. In this study, the LR model has only one case and does not utilize any other parameters for tuning.

Figure 7.

Support vector machine results (Source: Author’s own calculation)

Figure 8.

Decision tree results. (Source: Author’s own calculation)

Figure 9.

Decision tree results (Source: Author’s own calculation)

For SVM, although it has higher computational costs, it achieves better results compared to LG based on the highest MCC attained in both B Score and C Score datasets. Overfitting occurs with SVM, especially in the C Score dataset, where the MCC on the evaluation set tends to be contrary to the training set. In terms of trends, F-Recall and F-Precision follow a similar pattern to MCC. However, in terms of magnitude compared to LGR, there is not much improvement, fluctuating around 60% (B Score) and 50% (C Score).

Note: In each graph, the model cases are arranged in increasing order of the evaluation criteria on the training set. Therefore, the order of the model tuning cases will differ in the graphs. A case that achieves the highest MCC value may not necessarily have the highest F-Recall or F-Precision value.

Furthermore, it can be observed that in SVM, if default parameters are used, the evaluation criteria on the validation set are not optimal.

For DT models, in the case of B Score, the MCC on the evaluation set shows a significant improvement through different parameter cases. In most cases, over-fitting does not occur, except for around 10% of the cases where the MCC on the training set increases faster than on the evaluation set. This is somewhat contrary to the F-Recall criterion, where approximately 10% of the cases on the training set shows a rapid increase but a slight decrease on the evaluation set. Note that these 10% of cases are not completely overlapping.

In the case of C Score, both MCC and F-Precision tend to decrease significantly on the evaluation set despite a rapid increase on the training set. Figure 19 clearly demonstrates the inverse relationship between the training and evaluation sets for both MCC and F-Precision criteria. Therefore, choosing an optimal case for DT requires a trade-off between accuracy on the training and evaluation sets.

In the DT model, it can be observed that when using default parameters, the results on the training set are the best. However, these results are not consistent with the evaluation set. Except for tuning based on MCC for B Score customers, the default parameters yield the best results on both the training and evaluation sets.

The results in RF and DT are quite consistent with each other in terms of the trends of evaluation criteria for both B Score and C Score. In the case of B Score, the highest MCC achieved on the evaluation set is significantly higher in RF compared to DT. However, when considering F-Recall, the difference between the two models is not significant.

Similarly to the other models, overfitting in RF is not a major issue on the B Score, but it becomes more pronounced on the C Score as the accuracy on the training set decreases the accuracy on the evaluation set. When using the default parameters for the RF models, the results are also similar to the DT models.

In addition to evaluating the adjusted model cases based on the training and testing sets, it is necessary to assess the relationship between MCC and F-Recall, as well as F-Precision, to ensure the selection of appropriate criteria.

According to Figure 10, all criteria are represented on the evaluation set. When arranging the cases in increasing order of MCC, there is a clear positive correlation between F-Precision and the C Score dataset. However, on the B Score dataset, there is no clear relationship. This leads to the following consequences:

Figure 10.

MCC versus F-Recall and F-Precision (Source: Author’s own calculation)

In the case of DT and RF models, using the default parameters would be the best option when evaluating the models based on the MCC criterion.

According to the results in Table 5, for the B Score dataset, the choice of evaluation criterion has an impact on the selected model. Specifically, if MCC is used, RF should be chosen, and if F-Recall is used, SVM is the best model.

The use of MCC demonstrates consistency in both the training and evaluation sets. The RF model shows the best results across all evaluation criteria, with an overall accuracy of 81.84%. However, considering the importance of minimizing the error in identifying customer risk, if SVM is chosen based on the F-Recall index, the overall accuracy drops to 46.62%. The RF alert system correctly identifies only 67.45% of customers who actually become delinquent, while SVM achieves 91.48%, meaning that 32.55% (with RF) or only 8.52% (with SVM) of at-risk customers are overlooked. Additionally, from the perspective of alert effectiveness, 75.26% of customers receive accurate delinquency alerts, indicating that 24.74% of customers receive false alarms. In SVM, the rate of false alarms is more severe, with only 37.52% of customers receiving accurate alerts.

A comprehensive view of the selected models based on the F-Recall criterion reveals that RF has an F-Recall of 70.46%, which is not a significant improvement compared to SVM’s 71.04%. However, RF has relatively balanced Recall and Precision rates (70.78% and 69.20%, respectively), while the best-performing SVM model has a significant disparity (91.48% Recall and 37.52% Precision). Therefore, RF based on the F-Recall criterion is also a promising model. It should be noted that RF based on MCC and RF based on F-Recall will have different parameter sets.

In the dataset of customers who have become delinquent (C Score), the results in Table 6 are inconsistent across different evaluation criteria. It can be observed that tree-based models (DT and RF) yield higher results on the training set compared to other models. However, when considering the evaluation set, each model has different effectiveness based on different evaluation criteria.

Returning to the model selection for C Score, if MCC or F-Precision is used, the best model would be SVM, but with different parameter sets. Considering the importance of minimizing errors in identifying customers for credit improvement, it is advisable to prioritize evaluation criteria that emphasize the magnitude of Precision. In other words, incorrectly identifying a customer who can benefit from credit improvement can have more severe consequences than accurately predicting how many customers can actually improve their credit situation. Incorrect identification could lead the bank to implement support policies for the wrong customer segments.

Therefore, the optimal model for customers in the C Score dataset could be SVM based on MCC with a Precision of 58.44% and an F-Precision of 57.40%. Alternatively, SVM based on F-Precision could also be considered with corresponding figures of 62.3% and 57.93%.

In summary, the model selection for early warning system can be as follows, Table 7.

The results of the model evaluation during deployment period indicate the following:

Firstly, for the B Score dataset, when fine-tuning the model using the MCC criterion, the performance of the RF model is not particularly good when used with new data due to significantly lower Recall and F-Recall criteria compared to the evaluation set during the fine-tuning process. However, when using the models obtained from fine-tuning based on F-Recall, such as SVM and RF, there is a noticeable improvement in results, as shown in Table 8.

Overall, the effectiveness of real-world operations did not reach the level observed during the model evaluation in the tuning process. Most evaluation metrics decreased, but it can be observed that with the current data, tuning should be based on risk management criteria such as Recall or F-Recall, rather than a general accuracy indicator like MCC, as the decrease in important metrics is less significant. Additionally, the RF model did not perform as stably as the SVM model based on the important criteria. It is important to note that although SVM has a high Recall of 91.53%, its Precision is only 22.44%, which indicates an overly cautious approach as many customers are predicted to be at risk of default, which may be relatively safer than using RF, which only identifies 38.15% of customers who will actually default.

Secondly, in the C Score customer group, the evaluation criteria also decreased in the test set, but the decrease was not as significant as in the B Score group. Once again, it can be seen that tuning based on the F-Precision criterion yields better results than MCC. Furthermore, for the B Score group, SVM models performed the most stable although the overall accuracy level is not high.

Table 9 shows that both important model evaluation criteria decreased in the test set, but the decrease was not as substantial as in the B Score customer group. Despite the decrease, tuning based on F-Precision still achieved a higher level of 44.91% compared to 40.33% for MCC.

This study provides evidence that various factors related to customer deposit behavior, and past loan characteristics can predict the likelihood of customer credit migration within one year. This was achieved through the development of two models: B Score, which predicts the likelihood of a customer transitioning to a delinquent group, and C Score, which predicts the potential for a customer in the delinquent group to improve their credit status.

The study applied four basic machine learning models: Logistic Regression, Support Vector Machine, Decision Tree, and Random Forest. These models were tuned by adjusting their parameters to improve prediction results. These tuned parameters have been demonstrated to be better than the default parameters.

Among these models, LR served as the baseline model, commonly used in banks due to its simplicity and interpretability. However, the other models showed significant improvement in accuracy after tuning. Specifically, LR achieved accuracies of 64.66% and 63.79% for B Score and C Score, respectively, while the tuned models achieved higher accuracies of 81.84% and 71.60%. This suggests that there are alternative algorithm choices that can be applied in the early warning system, replacing LR.

This study also investigated which evaluation criteria should be used to obtain an optimal model. Typically, models are tuned to maximize overall accuracy. However, this study highlights the importance of selecting appropriate criteria aligned with the significance of early warning alerts in credit risk management. Specifically, for B Score models, F-Recall should be chosen, while F-Precision should be prioritized for C Score models. These criteria emphasize caution in avoiding false alarms in credit risk management. Applying more general criteria, such as MCC, which considers all types of prediction errors, may lead to serious consequences in bank credit operations.

Besides, for the dataset used in this study and the applied tuning methods, the SVM model yielded the best results compared to the other models. Firstly, in the B Score customer set, although RFC achieved an impressive accuracy of 81.84% when tuned with MCC, it fell short in terms of criteria that prioritize caution in banking. RFC only achieved an F-Recall of 67.45%, which was lower than SVM’s 71.04%. Another tuned RFC model with F-Recall achieved only 70.46%. Secondly, in the C Score customer set, SVM also had the advantage in both tuning methods using MCC and F-Precision, with F-Precision values of 57.40% and 57.93%, respectively. Although these two numbers are close, in practical applications, there will be more noticeable differences.

Finally, the study also implemented an alert system on the test dataset, which was entirely new data not used in training the models and was not a randomly sampled subset of the model-building dataset. These new data were generated after the model was constructed, specifically after the cutoff date of December 2021. After the actual credit migration of customers, the study compared the results with the model’s predictions. The significant issue to acknowledge is that all evaluation criteria for the models decreased. In the B Score set, the accuracy criterion decreased to 78.67% and 76.56% for the two RFC models from the initial values of 81.84% and 79.85%, and more significantly, SVM dropped drastically from 46.62% to 36.86%. However, when considering cautious criteria, SVM’s Recall remained stable at 91.48%, and F-Recall reached 71.04%, higher than both RFC models at 68.88% and 70.46%. For the C Score set, the two optimized SVM models also experienced a decrease in Precision and F-Precision values. As mentioned earlier, although these SVM models had approximate prediction values, the model with a slightly higher F-Precision would perform better on the test dataset. Thus, even though the models did not perform optimally on the test dataset, the cautious approach was still ensured to avoid losses for the bank.