Remote Sensing Image Object Detection Method Based On Improved YOLOv3

The backbone network of YOLOv3 adopts Darknet-53 network, in which the average pooling layer, full connection layer and Softmax are removed, and is mainly composed of convolution and residual modules. An important feature of Darknet-53 is that Residual network is used. Residual convolution in Darknet-53 is carried out first with a convolution kernel size of 3×3, the convolution with step size 2 will compress the width and height of the input feature layer. At this time, a feature layer can be obtained. After the feature layer is named layer, the 1×1 convolution and 3×3 convolution are carried out for the feature layer, and the result is added to the layer, and then the residual structure is formed Through constant 1×1 convolution and 3×3 convolution and superposition of residual edge, can greatly deepened the characteristics of the residual network is easy to optimize, and to improve accuracy to by adding considerable depth. Its internal residual block uses the jump connection, easing in the depth of neural network to increase the depth of gradient disappeared.

Each convolution part of Darknet-53 uses the unique DarknetConv2D structure. L2 regularization is performed during each convolution, and Batch Normalization is performed after completion of convolution with Leaky ReLU. The normal ReLU is to set all negative values to zero, Leaky ReLU gives all negative values a non-zero slope which can be mathematically expressed as: (1) yi={xixi≥0xiaixi<0 {{\rm{y}}_i} = \left\{ {\matrix{ {{x_i}} \hfill & {{x_i} \ge 0} \hfill \cr {{{{x_i}} \over {{a_i}}}} \hfill & {{{\rm{x}}_i} < 0} \hfill \cr } } \right.

The process of obtaining prediction results from features can be divided into two parts: building FPN [16] feature pyramid to enhance feature extraction; Three effective feature layers were predicted by YOLO Head.

In the feature utilization part, YOLOv3 extracts multiple feature layers for target detection. There are altogether three feature layers extracted. Three feature layers are located in different positions of main Darknet-53, namely middle layer, middle and lower layer, and bottom layer. The shapes are (52,52,256), (26,26,512), (13,13,1024). After obtaining three effective feature layers, the three effective feature layers can be used to construct the FPN layer. The construction method is as follows: the feature layer of 13×13×1024 is convolution processed for 5 times. After processing, YOLO Head is used to obtain the prediction results, and part of it is used for up sampling UpSampling2d and 26×26×512 features. The shape of the combined characteristic layer is (26,26,768). The convolution processing is performed again for 5 times in combination with the feature layer. After the processing, YOLO Head is used to obtain the prediction results, and part of up sampling UpSampling2d is used to combine with 52×52×256 feature layer. The shape of the combined feature layer is (52,52,384). Five convolution processes were carried out in combination with the feature layers. After the processing, the prediction result of feature pyramid was obtained by YOLO Head. Feature fusion of feature layers of different shapes was beneficial to extract better features.

Three enhanced features can be obtained by using FPN feature pyramid, whose shapes are (13,13,512), (26,26,256), (52,52,128), and then the feature layers of these three shapes are passed into YOLO Head to obtain the prediction result YOLO Head is essentially a 3×3 convolution plus a 1×1 convolution, 3×3 convolution is for feature integration, 1×1 convolution is for adjusting the number of channels.

The shape of the output layer is (13,13,75), (26,26,75), and (52,52,75). The last dimension is 75 because the graph is based on VOC dataset. It has 20 classes, and YOLOv3 is for each feature layer There are 3 prior boxes for each feature point, so the number of channels for the prediction result is 3×25. If coco training set is used, there are 80 kinds of classes, and the last dimension should be 255=3×85. Shape of the three feature layers is (13,13,255), (26,26,255), (52,52,255). The actual situation is that N×416×416 images are input, and three shapes of (N,13,13,255), (N,26,26,255) and (N,52,52,255) will be output after multi-layer operation, corresponding to each figure is divided into 13×13, 26×26 and 52×52 grid Position of 3 prior boxes on.

According to the second step, the prediction results of three feature layers can be obtained, and shapes are: (N,13,13,255), (N,26,26,255), (N,52,52,255), Each effective feature layer divides the whole image into grids corresponding to its length and width. For example, the feature layer (N,13,13,255) divides the whole image into 13×13 grids. Then several prior boxes are built from each grid center, these boxes are pre-configured boxes for the network, the network's predictions determine whether these boxes contain objects and what kind of object it is. Since each grid point has three prior boxes, the prediction result above can be reshape into: (N,13,13,85), (N,26,26,3,85), (N,52,52,3,85). Where 85 can be split into 4+1+80, where 4 represents x_offset, y_offset, h and w the four, the parameters of the prior box 1 represents a priori whether the box contains objects, 80 represent the types of the a priori box, because coco got the 80 class, so there is 80 but this result does not correspond to the final forecasting box in the picture, you also need to decode can complete YOLOv3 decoding process is divided into two steps:

Add x_offset and y_offset to each grid point, and the result is the center of the prediction box.

PaNet(Path Aggregation Network) is the addition of a bottom-up Path Augmentation to FPN. You can see in Figure 2 (a) the FPN is top-down route, through the lateral connection, transfer the strong semantic characteristics of the top down, only to enhance the characteristics of the pyramid semantic information For example, when the underlying characteristics to the P5 (red line), after very multilayer networks among them, at this point at the bottom of the target information is very fuzzy, so the FPN extensions, added a bottom-up route (green route, bottom ->P2-> N2~N5, where the path passes through less than 10 layers), as shown in Fig. 2 (b), thus compensating and reinforcing the positioning information.

Figure 2.

By applying PaNet to YOLOv3, another round of down sampling is conducted on the basis of its FPN structure to Further strengthen feature fusion, as shown in Fig. 3, thus improving the accuracy of YOLOV3 for detecting dense small targets and multi-scale targets in remote sensing images.

Figure 3.

SimOTA's function is to set different positive sample numbers for different targets, such as ants and watermelon. Traditional positive sample allocation schemes usually assign the same positive sample numbers to watermelon and ants in the same scene, so either ants have many low-quality positive samples, or watermelon only has one or two positive samples. It's not appropriate for either way of distribution. Therefore, SimOTA firstly calculates a cost matrix, which represents the cost relationship between each real box and each feature point. The cost matrix consists of three parts:

The higher the degree of overlap between each real box and the prediction box of the current feature point, it means that this feature point has tried to fit the real box, so its Cost will be smaller.

The process application of SimOTA is as follows:

Calculate the coincidence degree between each real box and the current feature point prediction box.

All experiments were carried out on a Linux operating system computer equipped with AMD R9-5900HX CPU, Nvidia 2080Ti GPU, 16G video memory and 16G memory. The remote sensing image dataset used in the experiment is DOIR dataset, which contains 23463 remote sensing images and 192472 object instances. These instances are manually marked as boundary boxes of axial pairs, covering 20 common object categories. The size of data set images is 800×800 pixels, and the spatial resolution is 0.5m~30m Among them, the training set accounted for 1/2, the verification set accounted for 1/6, and the test set accounted for 1/3.

The evaluation indicators of the experiment include precision(P), recall (R), average precision (AP), and mean Average Precision of all categories(mAP). The calculation formula of each index is as follows: (2) R=TPTP+FN {\rm{R}} = {{TP} \over {TP + FN}} (3) P=TPTP+FP P = {{TP} \over {TP + FP}} (4) AP=∫01Psmooth(r)dr AP = \int_0^1 {{P_{smooth}}\left( r \right){d_r}} (5) mAP=∑j=1KAPiK mAP = {{\sum\nolimits_{j = 1}^K {A{P_i}} } \over K}

Where TP, FP, TN and FN represent positive samples with correct predictions and positive samples but wrong predictions and negative samples with correct predictions and negative samples with wrong predictions. P_smooth(r) is a smooth P-R curve.

The comparison results of YOLOv3 and the improved YOLOv3 in this paper are shown in TABLE I. It can be seen that the detection accuracy of the improved YOLOv3 for small objects has been significantly improved, and the mAP has increased by 15.1 points. In terms of single category, the improved method is applied to the Wind mill, Airplane, Tennis Court, Expressway service area, Baseball fields and other types of small object detection performed better. The best-performing ballpark had an accuracy of 98.2 percent.

P-R curves of each category are shown in Fig. 4, and the detection effect of the algorithm in this paper on small targets in remote sensing images is shown in Fig. 5, from which it can be seen that the improved YOLOv3 has a better detection effect on small targets.

Figure 4.

Figure 5.