Deep CNN and twin support vector machine based model for detecting potholes in road network

This layer is responsible for collecting the data from the asphalt road, either in the context of pothole or no pothole. The real-world road images are captured using the smart camera in daytime. These images are labeled either pothole road images or no pothole road images. The final dataset contains 650 images with latitude and longitude information. A local folder is created on the system to store these images.

This subsection accomplishes the different preprocessing tasks associated with the road images. A Python environment is chosen to perform the preprocessing tasks. Initially, the images are resized using the morphological operations. The image resizing is done by using the affine transformation. The size of the initial images is 512 × 512, and it can be reduced to 64 × 64 using the affine transformation. It is also stated that most of the asphalt images have black color, but the present study considers the grayscale images for pothole detection. Hence, all road images are converted into grayscale images in which pothole is highlighted with black color and the rest of the area is highlighted with white color. Furthermore, the canny function is adopted to compute the edges of the pothole, and the region of interest is determined using Kapoor entropy. ROI is considered to extract the relevant features in the feature extraction phase. In the proposed model, the morphological operations are considered in the data-preprocessing layer, and the aim of these operations is to determine the shape or structure of features in an image. These operations are applied on the grayscale images of potholes to enhance or extract structural components. In pothole detection, erosion, dilation, opening, and closing are used. Erosion removes small white noises and shrinks the boundaries of foreground regions, and also helps eliminate irrelevant features. On the contrary, dilation can enlarge the white regions (e.g., pothole regions after thresholding), which helps in connecting broken parts of the detected pothole contour. Opening is useful for removing small objects or noise while preserving the shape of larger objects, making it ideal for cleaning up preprocessed images. Closing helps in filling small holes or gaps within the pothole region, which improves contour integrity. These operations are particularly useful before edge detection or contour extraction as they enhance the structural consistency of potholes in the image, facilitating accurate feature localization. Affine transformations are a set of geometric transformations that preserve lines and parallelism in pothole images. It includes a diverse set of operations like scaling, rotation, translation, shearing, and reflection. In pothole detection, scaling is especially important for resizing images to a uniform resolution while maintaining aspect ratios, which ensures consistent input dimensions for deep learning models. Translation is used to align features spatially, which is helpful when merging datasets with varying coordinate origins. Rotation helps the model become invariant to different orientations of potholes on the road, thereby improving generalizability. Additionally, shearing can be used for data augmentation by slightly distorting the images to simulate different camera angles and perspectives. These transformations not only standardize the data but also increase dataset diversity and enhance model robustness.

Kapoor entropy is a generalized form of entropy that measures the uncertainty or randomness within an image. It can consider the pixel intensities and also the underlying structure and spatial distributions for measuring the uncertainty. Unlike the other traditional entropy measures, such as Shannon entropy, Kapoor entropy also provides a more quantification of information content by introducing a parametric component that adjusts sensitivity to variations in pixel intensities. In our approach, Kapoor entropy is employed to dynamically assess the information in different sub-regions of the preprocessed image. The regions with higher Kapoor entropy values are considered to have a higher degree of variability. It can be described as follows: (1) Hth1,th2,…,thn=H0+H1+…+HnH0=−∑j=0th1−1pjw0lnpjw0,w0=∑j=0j=0pjH1=−∑j=th1th2−1pjw1lnpjw1,w1=∑j=th1th2−1pjH0=−∑j=thnL−1pjwnlnpjwn,wn=∑j=thnL−1pjfKapurth1,th2,…,thn=argmaxHth1,th2,…,thn \matrix{ {{\rm{H}}\left( {{\rm{t}}{{\rm{h}}_1},{\rm{t}}{{\rm{h}}_2}, \ldots ,{{\rm{th}}_n}} \right) = {{\rm{H}}_0} + {{\rm{H}}_1} + \ldots + {{\rm{H}}_n}} \hfill \cr {{{\rm{H}}_0} = - \sum\limits_{j = 0}^{t{h_1} - 1} {{{{p_j}} \over {{w_0}}}ln{{{p_j}} \over {{w_0}}}} ,{w_0} = \sum\limits_{j = 0}^{j = 0} {{p_j}} } \hfill \cr {{{\rm{H}}_1} = - \sum\limits_{j = t{h_1}}^{t{h_2} - 1} {{{{p_j}} \over {{w_1}}}ln{{{p_j}} \over {{w_1}}}} ,{w_1} = \sum\limits_{j = t{h_1}}^{t{h_2} - 1} {{p_j}} } \hfill \cr {{{\rm{H}}_0} = - \sum\limits_{j = t{h_n}}^{L - 1} {{{{p_j}} \over {{w_n}}}ln{{{p_j}} \over {{w_n}}}} ,{w_n} = \sum\limits_{j = t{h_n}}^{L - 1} {{p_j}} } \hfill \cr {{{\rm{f}}_{{\rm{Kapur}}}}\left( {{{\rm{th}}_1},{{\rm{th}}_2}, \ldots ,{{\rm{th}}_n}} \right) = {\rm{argmax\;}}\left\{ {{\rm{H}}\left( {{{\rm{th}}_1},{{\rm{th}}_2}, \ldots ,{{\rm{th}}_n}} \right)} \right\}} \hfill \cr }

In this layer, a deep CNN model is considered to capture the relevant features from the road image dataset. To determine the relevant feature, this layer considers the segmented greyscale images as input. It is observed that the deep CNN model avoids the manual generation of the high-order features from the road image dataset. This model generates efficient and high-order features from the low-order feature inputs. Further, a feature extractor is designed with the help of the deep CNN model for extracting the effective features. This process is illustrated in Figure 2. By inspiring the wide popularity and applicability of the deep CNN model (Nogales & Benalcázar, 2023; Deepak & Ameer, 2022b; Khater & Gamel, 2023; Gagliardi et al., 2023), a feature extractor model is designed based on the small multiple convolutional kernels that are stacked to each other and can generate an extended receptive field. It is also assumed that such a network can increase the depth of the model and also non-linear transformations as compared to large convolutional kernels. It is also noticed that fewer parameters are required to learn the model. In our proposed feature extractor, 6 convolutional blocks are designed by combining the 13 convolutional layers. Further, the convolutional layers in a convolutional block produce the same quantity of feature maps and size of the output feature map, except last convolutional block. The last block consists of three cascading convolutional layers, while the rest of the blocks contain two stacked convolutional layers. These convolutional layers consider the 3 × 3 continuous convolutional kernel using stride 1. The feature maps for this work are defined as 8 × 8, 16 × 16, 32 × 32, and 64 × 64. Further, a max-pooling operation is applied with a 2 × 2 pixel window with stride 2. The task of this operation is to reduce the size of the feature map but preserve the information. The pooling operation also restricts the parameters as well as the computation of the proposed model. Finally, an average pooling operation is implemented to obtain the features using the input image size 64 × 64. When comparing the average pooling operation to the conventional fully connected layer, the average pooling is less prone to overfitting and does not require any additional parameters. Finally, the last layer of the last convolutional block considers the softmax function, and the task of this function is to map the extracted feature with the probabilities of each feature vector. The feature extractor model is trained using the multi-class logarithmic loss function, and it can provide the discriminative features. In the training phase, the model employs the structural entropy graphs in the average pooling layer to generate feature characteristics. Consequently, the present study builds a trained feature extractor model that can take the input from the deep CNN model and the output from the average-pooling layer, respectively. The logarithmic loss function is defined using Eq. (2). (2) Loss=−1N∑m=1N∑n=1MXm,nlog2pm,n {\rm{Loss}} = - {1 \over {\rm{N}}}\sum\limits_{{\rm{m}} = 1}^{\rm{N}} {\sum\limits_{{\rm{n}} = 1}^{\rm{M}} {{{\rm{X}}_{{\rm{m,n}}}}{{\log }_2}{{\rm{p}}_{{\rm{m,n}}}}} }

Figure 2:

Proposed deep CNN-based feature extractor model. CNN, convolutional neural network.

The aim of the detection layer is to identify the potholes. For the identification of the potholes, this layer considers the relevant features extracted by the previous layer. Further, the potholes are detected using the TPSVM classifiers. The original TPSVM framework constructs two non-parallel hyperplanes by solving two smaller quadratic programming problems and each involving fewer samples. While this enhances computational efficiency, it introduces challenges related to model generalization and margin robustness. In the context of road condition detection, we analyzed the influence of reduced training data size on classifier stability and accuracy. This implies that when TPSVM is efficient, a minimum subset threshold is necessary to maintain structural risk minimization and margin fidelity. It is noticed that the TPSVM classifier provides better state-of-the-art classification results for imbalanced dataset and is widely used in the literature for classification tasks. TPSVM is an extension of the TSVM that computes the separate hyperplane using the two non-parallel parametric hyperplanes. It can be achieved by using the two small-sized support vector machines (SVMs), which are described below. (3) fax=wa×x+ba=0; and fcx=wc×x+bc=0 {f_a}\left( x \right) = \left( {{w_a} \times x} \right) + {b_a} = 0;\;{\rm{and}}\;{f_c}\left( x \right) = \left( {{w_c} \times x} \right) + {b_c} = 0

Assume, a binary classification problem that can divide the data items d₁ and d₂ such as data item d₁ belongs to class a, while data item d₂ belong to class c in the given n-dimensional space (S). Suppose the matrix (M) in M^d₁×n denotes the data items belong to class a, the matrix (N) in N^d₂×n denotes the data items belong to class c. Now, the TPSVM that allocates the data in two classes must follow the Eqs (4) and (5). (4) wa×x+ba≥0 such that i=1,2,3,……, d1 \left( {{{\rm{w}}_{\rm{a}}} \times {\rm{x}}} \right) + {{\rm{b}}_{\rm{a}}} \ge 0\;{\rm{such\, that}}\;{\rm{i}} = 1,2,3, \ldots \ldots ,\;{{\rm{d}}_1} (5) wc×x+bc≥0 such that i=1,2,3,……, d2 \left( {{{\rm{w}}_{\rm{c}}} \times {\rm{x}}} \right) + {{\rm{b}}_{\rm{c}}} \ge 0\;{\rm{such\, that}}\;{\rm{i}} = 1,2,3, \ldots \ldots ,\;{{\rm{d}}_2}

Further, the pair of the parametric margin of the hyperplane is obtained using the following constraint-based optimization problem, which is mentioned in Eqs (6) and (7). (6) minwa,ba,ξ 12wa2+p1d2e2TNwa+e2ba+v1d1e1Tξs.t. Mwa+e2ba≥0−ξ such that ξ>0 e1 \matrix{ {\mathop {\min }\limits_{{{\rm{w}}_{\rm{a}}},{{\rm{b}}_{\rm{a}}},\xi } {\rm{\;}}\left( {{1 \over 2}{{\left\| {{{\rm{w}}_{\rm{a}}}} \right\|}^2} + {{{{\rm{p}}_1}} \over {{{\rm{d}}_2}}}{\rm{e}}_2^{\rm{T}}\left( {{\rm{N}}{{\rm{w}}_{\rm{a}}} + {{\rm{e}}_2}{{\rm{b}}_{\rm{a}}}} \right) + {{{{\rm{v}}_1}} \over {{{\rm{d}}_1}}}{\rm{e}}_1^{\rm{T}}\xi } \right)} \hfill \cr {{\text{s.t.}}\;\left( {{\rm{M}}{{\rm{w}}_{\rm{a}}} + {{\rm{e}}_2}{{\rm{b}}_{\rm{a}}} \ge 0 - \xi } \right)\;{\rm{such\, that}}\;\xi > 0\;{{\rm{e}}_1}} \hfill \cr } and (7) minwc,bc,η 12wc2+p2d1e1TNwc+e1bc+v2d2e2Tηs.t. Mwc+e1bc≥0−η such that η>0 e2 \matrix{ {\mathop {\min }\limits_{{{\rm{w}}_c},{{\rm{b}}_c},\eta } {\rm{\;}}\left( {{1 \over 2}{{\left\| {{{\rm{w}}_{\rm{c}}}} \right\|}^2} + {{{{\rm{p}}_2}} \over {{{\rm{d}}_1}}}{\rm{e}}_1^{\rm{T}}\left( {{\rm{N}}{{\rm{w}}_{\rm{c}}} + {{\rm{e}}_1}{{\rm{b}}_c}} \right) + {{{{\rm{v}}_2}} \over {{{\rm{d}}_2}}}{\rm{e}}_2^{\rm{T}}\eta } \right)} \hfill \cr {{\text{s.t.}}\;\left( {{\rm{M}}{{\rm{w}}_{\rm{c}}} + {{\rm{e}}_1}{{\rm{b}}_c} \ge 0 - \eta } \right)\;{\rm{such\, that}}\;\eta > 0\;{{\rm{e}}_2}} \hfill \cr }

In Eqs (5) and (6), ξ and η can be described as the slack variables. The variables p₁,p₂ > 0; v₁,v₂ > 0 are defined as regularization parameters that are utilized to compute the penalty weight. e₁ and e₂ are defined as the vectors of appropriate dimension. Finally, Eq. (7) is used to allocate the data object (o) that belongs to either class a or b. (8) argmink=1,2wk.x+bkwk \mathop {\arg \min }\limits_{{\rm{k}} = 1,2} {{\left| {{{\rm{w}}_{\rm{k}}}.{\rm{x}} + {{\rm{b}}_{\rm{k}}}} \right|} \over {\left\| {{{\rm{w}}_{\rm{k}}}} \right\|}}

Apart from above, the regularization parameters in TPSVM serve key roles such as p₁ and p₂ penalize misclassified points and control the tradeoff between margin maximization and training error for each hyperplane. While, v₁ and v₂ scale the contributions of each class to prevent class imbalance from biasing the optimization process.

This subsection describes the performance parameters that are used in the present study. It is noticed that the research community widely adopts these parameters for evaluating the performance of the pothole detection models. The description of these parameters are summarized as

Accuracy rate is described as the correctly classified images with respect to the total number of images can be defined as the number of images. It can be evaluated using Eq. (9). (9) Accuracy=TP+TNTP+FP+TN+FN×100 {\rm{Accuracy}} = {{{\rm{TP}} + {\rm{TN}}} \over {{\rm{TP}} + {\rm{FP}} + {\rm{TN}} + {\rm{FN}}}} \times 100

In Eq. (7), TP is true positive, TN is true negative, FP is false positive, and FN is false negative. TP and TN describe the correctly classified images. Whereas incorrectly classified images are described by FP and FN.

Recall parameter gives the number of correctly classified images with respect to all positive labeled images. It is evaluated using Eq. (10). (10) Recall=TPTP+FN×100 {\rm{Recall}} = {{{\rm{TP}}} \over {{\rm{TP}} + {\rm{FN}}}} \times 100

Precision gives correctly classified images with respect to all correctly classified images (TP and FP). It is determined using Eq. (11). (11) Precision=TPTP+FP×100 {\rm{Precision}} = {{{\rm{TP}}} \over {{\rm{TP}} + {\rm{FP}}}} \times 100

F1-score can be described as a balanced parameter that can be used to evaluate the efficacy of the model, and it is computed using precision and recall. Eq. (12) is used to compute the F1-score. (12) F1−Score=2×Precision×RecallPrecision+Recall {\rm{F}}1 - {\rm{Score}} = 2 \times {{{\rm{Precision}} \times {\rm{Recall}}} \over {{\rm{Precision + Recall}}}}

The experimental results of the proposed deep CNN-TPSVM model and other existing techniques are discussed in this section. Several well-known parameters, such as accuracy, precision, recall, and F1-score, are considered for evaluating the efficiency of the proposed deep CNN-TPSVM model. Before evaluating the simulation parameters, the confusion matrix is also computed for each technique. Before proceeding the experimentation, the pothole image dataset is divided into a training set and a validation set. Further, 70% of the pothole images dataset is considered for the training set, while the rest of the images dataset is considered for the validation set. The present study also computes the accuracy and loss rates of the training and validation sets. AUC parameter is also considered to evaluate the efficacy of the proposed deep CNN-TPSVM model along with the rest of the techniques. Figure 3 depicts the confusion matrix of the proposed deep CNN-TPSVM model. It is also mentioned that the pothole dataset consists of 1,150 images, in which 860 images are labeled as pothole images and the rest of the images are labeled as no pothole images. Figure 3 illustrated that the proposed deep CNN-TPSVM model accurately identifies 841 pothole images out of 860 images. While the proposed model also identifies 16 images as pothole images, but in actual these images can be labeled as no pothole images. Figure 4 depicts the confusion matrix of the DBN, Inceptionv3, Vgg16, VGG19, ANN, and SVM techniques. It is observed that the DBN model identifies the pothole images more accurately among these models.

Figure 3:

Depicts the confusion matrix of proposed deep CNN-TPSVM model. CNN, convolutional neural network; TPSVM, twin parametric support vector machine.

Figure 4:

Depicts the confusion matrix of other existing models.

The experimental results of the proposed deep CNN-TPSVM model are illustrated in Table 1. The results are evaluated based on accuracy, F1-score, recall, and precision parameters. The results are also compared with popular techniques existing in the literature such as ANN, VM, VGG16, VGG19, Inceptionv3, and DBN. By analyzing the results, it is found that the proposed deep CNN-TPSVM model achieves a higher accuracy rate (96.12%) compared with other existing techniques. It is also seen that ANN exhibits a lower accuracy rate (79.73%) among all techniques. By analyzing the variants of the CNN models (VGG16, Vgg19, InceptionV3, and DBN), it is reported that the DBN model obtains higher accuracy (93.04%) among these variants. The proposed deep CNN-TPSVM model also obtains good results in terms of precision and recall parameters compared with other techniques. The proposed model has higher recall and precision rates (98.13% and 97.79%). Whereas, recall and precision rates of the DBN model are 94.30% and 96.31%, respectively. It is also stated that the precision and recall rates (85.82% and 87.32%) of the ANN technique are lower among all. Moreover, the F1-score parameter is also considers to validate the performance of the proposed deep CNN-TPSVM model. This parameter can be acted as a balance parameter that considers the only positive instance of the dataset. Hence, this parameter is evaluated using the TP, TN, and FP. Hence, this parameter specifies the goodness of the model and is more significant than the accuracy parameter. F1-score parameter results are also summarized in Table 1. It is seen that the proposed deep CNN-TPSVM model gets a higher F1-score rate (97.96%) than other techniques. On the contrary, ANN obtains a lower F1-score rate (86.56%) among all techniques. It is also stated that the proposed deep CNN-TPSVM model obtains better F1-score rate than an accuracy rate for the detection of potholes. Hence, it is said that the proposed model is a more significant and effective mode for detecting the potholes in road network. Figure 5 presents the comparative results of all techniques based on the accuracy, precision, recall, and F1-score parameters. The results illustrate the efficiency of the proposed deep CNN-TPSVM model in terms of accuracy, recall, precision, and F1-score. It is also observed that the F1-score of the proposed CNN-TPSVM model is better than its accuracy rate. Hence, it is said that the proposed deep CNN-TPSVM model is more prominent than other models.

Figure 5:

Comparative analysis of the results using proposed deep CNN-TPSVM model and existing models based on different performance parameters. CNN, convolutional neural network; TPSVM, twin parametric support vector machine.

The present study also considers accuracy and loss rates of the training and validation sets. The accuracy rate of the training and validation sets, along with epoch, is depicted in Figure 6. The accuracy rate of both sets is represented using a different color scheme. It is found that the accuracy rate of validation set is higher than the training set. The accuracy rate of the validation set is 0.958, whereas the accuracy rate of the training set is 0.921. The loss rate of the training and validation sets is depicted in Figure 7. It is seen that the loss rate of the validation set is lower than the training set. The loss rate of the validation set is 0.042, whereas the loss rate of the training set is 0.079. The accuracy and loss rates of the training and validation sets are used to understand the overfitting issue of the model. So, it is found that the loss rate of the validation set is lower than the training set. Hence, the proposed deep CNN-TPSVM model cannot over fit the data, and the overfitting issue of the binary classification problem is successfully handled by the proposed model. In the present study, the AUC parameter is also computed for the proposed deep CNN-TPSVM model and other techniques. The main reason for the selection of the AUC parameter is the nature of the pothole dataset, as the pothole dataset contains imbalanced data, i.e., 860 images of pothole and 260 images of no pothole classes. In case of imbalanced dataset, the AUC parameter is one of the significant parameters for evaluating the efficiency of the model. It can be computed by TP and FP rates. The results of the AUC parameter are illustrated in Figure 8, and it is stated that the proposed deep CNN-TPSVM model obtains a better AUC rate compared with other techniques.

Figure 6:

Demonstrates the accuracy rate of the proposed deep CNN-TPSVM model based on training and validation sets. CNN, convolutional neural network; TPSVM, twin parametric support vector machine.

Figure 7:

Demonstrates the loss rate of the proposed deep CNN-TPSVM model based on training and validation sets. CNN, convolutional neural network; TPSVM, twin parametric support vector machine.

Figure 8:

Depicts the AUC of proposed deep CNN-TPSVM model and other existing techniques. CNN, convolutional neural network; TPSVM, twin parametric support vector machine.