Backpack detection model using multi-scale superpixel and body-part segmentation

In most computer vision systems, smoothing is a step commonly used to remove noise. The median filtering method is a smoothing method that improves image quality in the spatial domain. This method is non-linear filtering and works almost the same way as mean filtering [18]. Another step that can also improve video quality is contrast enhancement, which aims to provide a better contrast distribution. The histogram equalization method is widely used in image contrast enhancement [19]. This method usually increases the contrast value of the image globally, especially if the contrast value in the image is located at a close distance. Eq. (1) is used to perform histogram equalization operations on the image. (1) pni=nin {p_n}\left( i \right) = {{{n_i}} \over n} Where

p_n (i) = images histogram for pixel value i

The FD method detects changes in the image area by differentiating between two adjacent frames [20]. When an area contains pixels at moments t₁ and t₂, there is a differentiation between f₁(x, y) and f₂(x, y). Differentiation is obtained by subtracting the k-th frame (f_k (x, y)) from the k-1 frame (f_k−1 (x, y)), as shown in Eq. (2). (2) Dx,y=fkx,y−fk−1x,y D\left( {x,y} \right) = \left| {{f_k}\left( {x,y} \right) - {f_{k - 1}}\left( {x,y} \right)} \right| The area resulting from frame differentiation can be the movement of objects, changes in lighting, or noise. Figure 2 shows the resulting image of the foreground detection process.

Figure 2:

Superpixel segmentation has been widely used in object detection methods [17], [21]. Simple Linear Iterative Clustering (SLIC) generates superpixels by clustering pixels based on their similarity in color and proximity [22]. SLIC is usually implemented in the labxy color space, where the clustering process is more manageable. The Lab channel is a vector of color pixels in the CIELAB color space, where it is widely recognized that the colors of the pixels that are close together are perceptually the same. Meanwhile, the xy channel is the pixel position in the image plane. The SLIC algorithm uses the K-Means clustering method, as shown below.

There are two features, namely the HOG and histogram features, used to classify images in this study. In the first scenario, training and testing of the model are carried out using the HOG feature only. In the second scenario, the two features (HOG and histogram) are combined and classified in a single matrix.

1.

HOG

The HOG feature classifies pixels based on orientation and direction [23]. In the HOG feature extraction process, several variables must be determined: orientation, pixels per cell, and cells-per-block. The orientations variable shows the number of bin orientations on the pixel gradient formed in the histogram. Pixels-per-cell is a variable that determines the number of pixels in rows and columns in each gradient histogram formed. The cell-per-block variable determines the area in which the number of histograms in the cells will be normalized. In a 64 × 128 image, we would have 8 × 16=128 cells. To create 16 × 16 blocks, 4 cells would need to be combined, resulting in 7 × 15=105 blocks. Each block has a vector 36 × 1 feature; hence, the total features for the image would be 105 × 36 × 1= 3780 features. Figure 3 shows a cell and a block in a 64 × 128 image.

Figure 3:

Feature dimensions that are too large would slow down the training and testing process. By contrast, a small feature dimension can decrease the execution time without unduly affecting the model's accuracy [25]. We adopted PCA (Principal Component Analysis) to reduce the variable dimensions by eliminating the correlation between the independent variables and transforming the original independent variables into new uncorrelated variables [26]. Let i be the vector feature of the image, and let h be its dimension. The projection matrix with the first P principal components is obtained by solving the eigen equations of the covariance matrix S, defined as (4) S=1N∑n=1N(xi−x¯)(xi−x¯)T S = {1 \over N}\sum\limits_{n = 1}^N {\left( {{x_i} - \bar x} \right){{\left( {{x_i} - \bar x} \right)}^T}} The mean of the feature vector of the training sample is defined as (5) v¯=1N∑i=1Nxi \bar v = {1 \over N}\sum\limits_{i = 1}^N {{x_i}} The final descriptor is computed using the projection matrix as y = U^T(x − x̅).

Boser, Guyon, and Vapnik developed the Support Vector Machine (SVM) method, which was first presented in 1992 at the Annual Workshop on Computational Learning Theory [27]. The concept of SVM can be explained simply as an attempt to find the best hyperplane that separates two classes in the input space. The available data is denoted as x_i ∈ R^d, while labels are denoted as y_i ∈ {−1,+1} where i = 1,2,… ..l, and l is the amount of data. In our task, y_i = 1 can be considered to be the backpack, and y_i = −1 can be classified as the body or background. We assumed that the two classes could be separated perfectly by a hyperplane d, which is defined as w→.x→+b=0 \vec w.\vec x + b = 0 . The training feature x_i, belonging to class −1, can be formulated as a training feature that satisfies the inequality w→.x→+b≤−1 \vec w.\vec x + b \le - 1 , while the training feature x_i, belonging to class +1, can be formulated as w→.x→+b≥+1 \vec w.\vec x + b \ge + 1 . The distance of the hyperplane to the closest points in either class is 1w→ {1 \over {\left| {\vec w} \right|}} ; then the combined distance can be determined 1w→+1w→=2w→ {1 \over {\left| {\vec w} \right|}} + {1 \over {\left| {\vec w} \right|}} = {2 \over {\left| {\vec w} \right|}} . Then, to optimize this distance in a large dataset, the Lagrangian multiplier can be used, as shown in Eq. (6). (6) Lw→,b,a=12‖w→‖2−∑i=1laiyixi→.w→+b−1i=1,2,3,…,l \matrix{{L\left( {\vec w,b,a} \right) = {1 \over 2} \left\| {w{^2}} \right\| - \sum\limits_{i = 1}^l {{a_i}\left( {{y_i}\left( {\left( {\overrightarrow {{x_i}} .\vec w + b} \right) - 1} \right)} \right)} } \hfill \cr {\left( {i = 1,2,3, \ldots ,l} \right)} \hfill \cr }

If the input data x_i cannot be linearly separated, then it can be mapped into a higher dimensional feature space k by utilizing the kernel. There are three types of kernels in SVM: Linear, Polynomial, and Radial Basis Function (RBF). The kernel function used in this study is the linear kernel, which is denoted by k(x_i, x_j)=x_i^Tx_j. The linear kernel is quite effective, and its required processing time is also faster [28].

The body-part method spatially localizes the CO area at a specific location based on body proportions and camera position [6]. Various kinds of CO can be detected by utilizing the general position of the CO carried based on the bend line.

In Figure 4, if h is the height of the head, then it can be derived that the height and width of the human body are H = 8h and W = 2h, respectively. The horizontal diameter of the body is denoted as C, and the vertical diameter of the body is denoted as B. If T is the highest point of the head and L is the leftmost point of the body, then Eq. (7) can be used to obtain B, and Eq. (8) can be used to obtain C. (7) B=T+4h B = T + 4h (8) C=L+H C = L + H

Figure 4:

Based on observations of the dataset we use at the training stage, generally, the backpack is located around or slightly above the bend line. Therefore, only those superpixels located above the bend line are used in the classification process. Based on the human body proportion model, the bend line can be calculated using the head area.

To detect the head area, we first collected head segments and then extracted their features (Figure 5). The obtained feature vectors were then used to train the head detector model. In the testing phase, the trained model was then used to detect head segments, and after the head was obtained, the bend line was then generated using Eq. (7) and Eq. (8).

Figure 5:

The bend line is then used as a reference for selecting the superpixels that will be used as ROI to detect backpacks. Figure 6 shows the training process for the body-part model and superpixel selection based on the bend line.

Figure 6:

We built the DIKE20 dataset to train the model we used for the classification process. Each acquired pedestrian image was then selected manually, where only pedestrians carrying backpacks were used to train the model. The DIKE20 dataset was acquired using four cameras. Each camera was mounted on a wall of approximately 3 meters. Camera A was used to record objects from the front view, cameras B and C were used to record objects from the side view, and camera D was used to record objects from the rear view.

Figure 7 shows the camera configuration in the acquisition room. Another objective was for the image to present the pedestrian body intact, and not overlapping with pedestrians or other objects. There were 966 images selected from various views used to train the model.

Figure 7:

The testing process of the trained model was carried out on three datasets: DIKE20, PETS2006, and i-LIDS. In the DIKE20 dataset, 271 images were selected. This study used all 7 of the scenarios in the PETS2006 dataset. The images in each scenario were selected based on the constraints previously mentioned. 65 images were selected from the first scenario, 70 from the second, 33 from the third, 12 from the fourth, 55 from the fifth, 66 from the sixth, and 22 from the seventh. In the i-LIDS dataset, there were six scenarios.

Only the first scenario was suitable for this research. This scenario was acquired using two cameras. From the first camera, 121 images were selected, and from the second camera, 64 images were selected. Table 1 shows the number of images used at the testing stage on each dataset.

Superpixels are generated using the K-Means algorithm on an image using 5-dimensional color space (Labxy). The variable k in the K-Means algorithm shows the scale, which is the number of superpixels to be formed. Due to the varying sizes of CO in this study, the segmentation was carried out on three scales: l₁, l₂, and l₃, which are 10, 15, and 20 superpixels respectively. Figure 8.a shows the segmentation result on the l₁ scale, Figure 8.b shows the segmentation result on the 1₂ scale, and Figure 8.c shows the segmentation result on the 1₃ scale.

Figure 8:

Backpacks are generally located above the bend line. This study trained a model that recognized the head area by classifying segments through its HOG feature. After the head area was obtained, the bend line was identified by adding the coordinates of the body's highest point with four times the area of the head. Segments located above the bend line had their features extracted. Figure 9 shows the process of determining the body's vertical (B) and horizontal (C) diameter. After the center lines B and C were obtained, the segments above line B were selected. Table 2 shows the selected superpixels and their location based on bend line. The selected segments’ features were extracted and then classified to detect whether there was a backpack among the segments.

Figure 9:

We extracted two features to train the backpack detection model: the HOG feature and the histogram. The evaluation of the model's performance was carried out in two scenarios. The first scenario (BP_SC1) only utilized the HOG feature, while the second scenario (BP_SC2) utilized the HOG and histogram features combined.

The initialization values for extracting HOG features were: orientation = 8, pixels-per-cell = 16x16, and cells-per-block = 2. The histogram feature was extracted by converting the image into a grayscale color space. Each color bin was then calculated in the gray degree range of 0 to 255. Figure 10 shows the example of features extraction.

Figure 10:

The testing stage was carried out using two scenarios: BP_SC1 and BP_SC2. The ROC curve for each scenario in each dataset has been generated using a different threshold.

In the ROC curve, there is an area called AUC (Area Under the Curve), which shows the ratio of true positive (y-axis) and false positive (x-axis). The AUC value obtained by the model for the DIKE20 dataset was the same for both scenarios: 0.82. Figure 11 shows the ROC curve for each scenario on the DIKE20 dataset. Model performance was also measured using precision, recall, and F1 scores. In the DIKE20 dataset for the BP_SC1 scenario, the precision, accuracy, and F1 scores were 46%, 79%, and 60%, respectively. In the BP_SC2 scenario, the precision increase was 6%, and recall score increased by only 1%. The F1 score for the BP_SC2 scenario was 63%, where the score increased by 3% compared to the F1 score for the BP_SC1 scenario. Table 3 presents the precision, recall, and F1 scores on the DIKE20 dataset.

Figure 11:

Benchmarking of the constructed model was carried out by comparing the precision, recall, and F1 scores with the model developed by [9], [16], and [17]. In addition, we also measured the AUC score on each dataset on each scenario. The AUC score on the PETS2006 dataset was 0.86 for the BP_SC1 scenario and 0.89 for the BP_SC2 scenario. There was an increase in the AUC value of 0.03 for the BP_SC2 scenario. The ROC curve for each scenario for the PETS2006 dataset is shown in Figure 12.

Figure 12:

The AUC scores on the i-LIDS dataset were 0.86 and 0.83 for the BP_SC1 and BP_SC2 scenarios, respectively. There was a decrease in the AUC value of 0.03 in the BP_SC2 scenario. The ROC curve of each scenario for the i-Lids dataset can be seen in Figure 13.

Figure 13:

On the PETS2006 dataset, the best precision score is obtained by [17], whereas the best recall score is 85%, and was obtained by the BP_SC1 scenario. The highest F1 score was obtained by BP_SC2 (69%). Table 4 shows each method's precision, recall, and F1 scores on the PET2006 dataset.

The experimental results on the i-LIDS dataset are presented in Table 5. For the i-LIDS dataset, the best precision score is obtained by Ghadiri et al. [17]. Our methods, BP_SC1 and BP_SC2, had significantly higher recall scores (90% and 95%, respectively). However, the F1 scores obtained by [17] and our method (BP_SC2) are comparable.