Infrared Weak and Small Target Detection Algorithm Based on Deep Learning

RT-DETR is the first real-time object detector in the DETR series and consists of four models. Different networks use different backbones, among which rtdetr-l uses HGNetv2-l as the backbone network, which has the best performance under the same conditions with fewer parameters and computational costs. The network structure is shown in Figure 1. The RT-DETR-L network structure consists of three parts: Backbone, Neck, and Decoder. The backbone network HGNetv2 consists of four HG Stage modules, each of which is mainly composed of HG Blocks. The backbone network extracts feature maps of three scales at different levels as the input of the hybrid encoder. The hybrid encoder is composed of two modules, the AIFI encoder and the CCFM feature fusion, in which the AIFI is still a multi-head transformer in essence, and only the deepest S5 feature layer is processed, and the F5 feature layer is finally output, which is used as the input of the CCFM module together with the S3 and S4 features, and the upper feature fusion and the lower feature fusion are carried out twice, respectively. For the fusion result, the anchor frame is selected through IOU-ware Query Selection, and the top-k300 is finally selected as the input into the decoder for prediction output.

Figure 1.

RT-DETR network structure

In view of the problems of low contrast, high noise and unclear details in infrared images, the detection performance of the model can be improved by selecting an appropriate algorithm for image enhancement before the model is processed.

Dynamic Detail Enhancement (DDE) [8]and MSR algorithm are two commonly used image enhancement techniques in the preprocessing of infrared images. DDE separates the base layer and the detail layer of the image through the filtering algorithm, and enhances the detail layer to achieve significant enhancement of the details. However, the brightness and contrast of the base layer are limited, and it is easy to amplify the noise while amplifying the details. The multi-scale MSR algorithm improves the brightness and global contrast of the image through digital transformation and smoothing. However, the processing effect at the level of detail is limited, in complex scenes, the improvement of local contrast is not obvious. Therefore, the combination of DDE and MSR algorithm can dynamically adjust and enhance the detail and clarity of the image according to the complexity of the image content.

The detailed design process of the algorithm is as follows.

Bilateral filtering is used to decompose the original image into basic component I_B, and the output is: 1IB=1Wq∑p∈SGS(p)∗Gr(p)∗Ip

where the ^I_p represents the original infrared image, the ^G_S represents the pixel value weight, the ^G_r represents the spatial distance weight, and the ^W_q represents the sum of the weights of each pixel value in the filter window, which is used for the normalization of the weights.

The detail component is the result of subtracting the base component from the original image, The formula is shown in the following formula.2ID=Ip−ID

The basic component contains the large-scale structure and lighting information of the image, and the local details are enhanced by MSR for the basic component ^R_Bo 3RB=MSR(IB)

Finally, the detail component is recombined with the enhanced base component to obtain the final infrared image.

In order to solve the problem that small and medium-sized targets are difficult to detect in infrared image object detection, the original rt-detr-l uses HGNetv2, which is essentially still a CNN network structure, and uses HGBlock to extract features through deep convolution operations. Therefore, an improved RT-DETR network model is designed, and the EMA attention mechanism module is added to the backbone network of RT-DETR, so that it can extract multiscale feature information that is rich in global context information and differentiated features, especially small target feature information.

Figure 2 shows how to add an EMA module.

Figure 2

EMA module

At the same time, in order to suppress the interference of background and noise in the infrared image, the attention of small targets is improved. The introduction of CAMixing convolution-attention module enables CCFM to suppress the interference of irrelevant information and improve the denoising performance when multi-scale feature fusion, which is conducive to enhancing the modeling of global and local features and improving the detection rate of small targets.

Add the CAMixing module to the network, as shown in Figure 3.

Figure 3.

CAMixing module

In RT-DETR, the encoder is only used in S5, and comparative experiments are used to verify that it not only helps to significantly reduce the computational effort and improve the computational speed, but also does not cause significant damage to the performance of the model [9]. Therefore, the addition of CAMixing to the two-way downward feature fusion with S5 helps to improve the detection accuracy and reduce the amount of computation.

The original IOU-ware Query Selection uses GIoU to optimize the query selection process of the prediction box, and introduces an external rectangle to reduce the loss between the real box and the prediction box of the large target. However, for small target detection, a slight shift in the target may cause a significant change in its position information. GIoU does not directly consider the aspect ratio difference between the prediction box and the real box, and cannot fully capture the subtle changes in position information. Therefore, Shape-IoU is used instead of GIOU for small targets, and the loss is calculated by paying attention to the shape and scale of the bounding box itself, so as to optimize the detection accuracy of small targets. The regression loss calculated by Shape-IoU is shown in the following formula.4LS=1−IOU+DS+0.5ΩS $${L_S} = 1 - IOU + {D^S} + 0.5{\Omega ^S}$$

Among them, IOU is the intersection and union ratio, S is the zoom factor, and its value is related to the size and number of targets in the dataset. and, ^DS is the distance loss, which is calculated as follows: 5ww=2×(wgt)s(wgt)s+(hgt)s 6hh=2×(hgt)s(wgt)s+(hgt)s 7DS=hh×(xc−xcgt)sc2+ww×(yc−ycgt)sc2

Where ^wgt, ^hgt are the width and height of the GT box, ^x_c, ^y_c, xcgt , ycgt are the coordinates of the center point of the prior box and the GT box. ww and hh are calculated from the coordinates as the weight coefficients of the horizontal and vertical directions, and the loss of ^DS can be adjusted by adjusting the value of s.

Ω^S is the loss of shape. It follows the formula of SIOU: 8ΩS=∑t=w,h(1−e−wt)θ $${\Omega ^S} = \mathop \sum \limits_{t = w,h} {\left( {1 - {e^{ - {w_t}}}} \right)^\theta }$$

Where w,h are the width and height of the prior frame, respectively, θ determines the size of the shape loss, and in order to avoid the shape loss accounting for a relatively heavy proportion of the overall loss and the position change of the prediction frame, the genetic algorithm takes the value of 4.

In the case of an infrared small target image, it is mainly composed of the background, and only a small part is occupied by the target. It is easier to learn the features of the background than the features of the target during training. Therefore, the ATFL loss function is used to replace the original VFL classification loss, which decouples the target from the background, and uses the adaptive mechanism to adjust the loss weight, forcing the model to allocate more attention to the small target features. The ATFL expression is as follows: 9{ −(λ−pt)−ln(pt)log(pt)pt≤0.5−(1−pt)−ln(pt)log(pt)pt>0.5

Where ^p_t represents the current average prediction probability value, and ^λ (>1) is the hyperparameter. ATFL is balanced by improving the adaptive factor in TFL ^γ–ln(P_t). The weights of the samples [10] increase the contribution of small targets while reducing the time consumption of adjusting hyperparameters.

Table I shows the experimental environment in this paper, which is based on the Ubuntu 18.04 operating system, the graphics card model is RTX2080Ti, and the memory is 16GB. The experiment basically uses the parameters recommended by RT-DETR, builds the model based on Python3 and Pytorch framework, and uses the standard SGD optimizer, with batch-size set to 8 and epochs set to 100.

The infrared aircraft small target dataset [11] used in this experiment includes a total of 22 annotated data folders, and the image content is mainly based on the ground background, sky background, multiple aircraft, aircraft distance, aircraft approaching, etc. A total of 12177 infrared images were selected, with an image resolution of 256*256, a channel count of 1, and a bit depth of 24. This dataset is widely used in tasks such as object detection and target tracking.

In this experiment, Precision (P), Recall (R) and Average Precision (AP) are mainly used as network evaluation indicators, and their mathematical expressions are as follows: 10precision =TPTP+FN

Where TP stands for True Positives, FP stands for False Positives, and Precision measures the proportion of instances that the model predicts to be positive. Used to evaluate the accuracy of instances predicted to be positive samples.

Recall, also known as recall, is the proportion of the sample predicted as positive to the predicted sample, and its mathematical expression is: 11recall =TPTP+TN

The mathematical expression for the average precision (AP) is: 12AP=∫P(R)dR

AP is used to evaluate the Precision-Recall curves of the model at different thresholds, and is the average value obtained by integrating the Precision-Recall curves. It measures the average accuracy of the model’s predictions at different thresholds.

In addition, in order to evaluate the processing effect of the image preprocessing algorithm, PSNR and SSIM similarity were used to distinguish the similarity of the images before and after processing, the information entropy (Entropy) was used to reflect the retention degree of image information, and the average gradient (AG) and edge intensity (EME) were used to evaluate the clarity of the image.

The objective evaluation method is used to evaluate the quality of the improved images, and the effectiveness of the improvement is verified compared with other algorithms.

The final enhancement is shown in Figure 4.

Figure 4.

Image enhancement effect

Table II lists the verification results.

In this paper, the Shape-IoU loss function is introduced into the network model [12]. which has a scale parameter whose size can be determined by optimizing the parameters, and the s parameter is finally set to 0.4 by comparing different parameters on the dataset. Comparative experiments are shown in Table III.

In order to verify the effectiveness of each improvement point of the network, based on the RT-DETR-L network, six sets of comparative experiments were carried out on the dataset. and the environment and parameter settings were uniform. The experimental results are shown in Table IV, and "√' indicates that the corresponding method was used. It can be seen from Table 4 that after adding the EMA and CAMixing modules and improving the loss function, the algorithm achieves the best detection accuracy, and the AP value is 3.2% higher than the original RT-DETR, which proves the effectiveness of the improved module in this paper. Figure 5 shows the AP variation curve of the improved method more intuitively. The detection fluctuates greatly due to the influence of the dataset, but the improved network detection accuracy is significantly stable and the accuracy increases. Figure 6 shows the detection results of the original network and the improved network, and the prediction score increases significantly, which verifies the effectiveness of the improved method and proves that the improved method is better for the detection of dense targets.

Figure 5.

AP change curve

Figure 6.

Improved detection results