Fine-grained Recognition of Ships Under Complex Sea Conditions

As aerial image is often affected by complex weather, such as fog, this factor will cause blind to the ships and affect the recognition in subsequent experiments. The atmospheric scattering model [16] was proposed by McCartney in 1976 based on the atmosphere. The scattering model is an important theoretical basis for the removal of fog. The model is shown in equation (1). (1) I(x)=J(x)t(x)+A(1−t(x)) I\left( x \right) = J\left( x \right)t\left( x \right) + A\left( {1 - t\left( x \right)} \right)

In equation (1), I (x) is an image with fog, J (x) is an image without fog, t (x) is atmospheric scattering rate, which reflects the ability of ray to penetrate the fog. A denotes atmospheric light intensity, and it is a constant vector.

In this paper, the method of guided filtering [17] is used to defog. The guided filtering obtains the result image based on the equation (1), including a guide image I, an input image P, and an output image q. The guide image I needs to be set in advance according to specific application, it can also be directly taken as the input image P. For the i^th pixel in the input image, the calculation method can be expressed as: (2) qi=akpi+bk, ∀i∈ωk {q_i} = {a_k}{p_i} + {b_k},\,\forall i \in {\omega _k}

In equation (2), i is i^th pixel label, a_k and b_k are linear coefficients, and ω_k is local neighborhood. By calculating the gradients on both sides of equation (2), ∇ q = a ∇ p can be obtained. When the input image p has gradients, the output q also has similar gradients, so that the feature of the ship edge can be preserved while defogging. The minimum cost function between the input image with fog and the output image is shown in equation (3). (3) E(ak, bk)=∑i∈ωk((akpi+bk−pi)2+εak2) E\left( {{a_k},\,{b_k}} \right) = \sum\limits_{i \in {\omega _k}} {\left( {{{\left( {{a_k}{p_i} + {b_k} - {p_i}} \right)}^2} + \varepsilon a_k^2} \right)}

In the above formula, we introduce ɛ constants to make a_k converge. We can know from equation (3): ak=1|ω|∑pi2−μkp¯kσk2+ε, bk=p¯k−akμk {a_k} = {{{1 \over {\left| \omega \right|}}\sum {p_i^2 - {\mu _k}{{\bar p}_k}} } \over {\sigma _k^2 + \varepsilon }},\,{b_k} = {\bar p_k} - {a_k}{\mu _k}

Where μ_k and σk2 \sigma _k^2 are the mean and variance of the guide image p in the ω_k window, |ω| is the number of pixels in the p¯k {\bar p_k} window, p¯k {\bar p_k} is the mean of input image p in window ω_k.

For each pixel, the final relationship between the output image q and the input image p is shown as equation (4). (4) qi=1|ω|∑k, i∈ωi(akpi+bk)=a¯ipi+b¯i {q_i} = {1 \over {\left| \omega \right|}}\sum\limits_{k,i \in {\omega _i}} {\left( {{a_k}{p_i} + {b_k}} \right) = {{\bar a}_i}{p_i} + {{\bar b}_i}}

In this paper, the fog image is used as the guide image to guide the filtering, so as to obtain the result image q after defogging, as shown in Figure 2.

Figure 2.

As aerial images are shoot at high altitude, the ship in images is relatively small so that affect the recognition accuracy. In addition, as samples are limited and the feature of ship cannot be fully extracted during training, which can result in false or missed detection. We use multi-scale samples, that is, set each image to multiple scales. The specific idea is to reduce the training samples to three scales (1024*1024, 512*512,256*256), as shown in Figure 3, it is equivalent to add large numbers of small ship samples for training in the training set. As the number of small target samples increase, that can ensure the network extracts the features of small target effectively, thus avoiding false or missed detections. The experiment shows that the multi-scale training can make the distribution of the target sizes of various classes of training more uniform, so that the trained network model can be more robust to multi-scale targets.

Figure 3.

To achieve high recognition accuracy of ship, the effective features of the ship have to be extracted firstly, and the feature extraction of the ship is closely related to the structure of the convolutional neural network for feature extraction. The Convolutional Neural Network [18,19,20] (CNN) is a feature extractor composed of convolutional layers and sampling layers. The advantage is that it does not require complex preprocessing. Feature extraction and pattern classification are completely put into a black box. Through continuous optimization to obtain the required parameters, CNN gives the desired classification at the output layer. The combination of convolutional layer and pooling layer will directly affect the effective feature extraction of ships. After a large number of experiments, the pre-trained ZF [21–22] network is chosen. Compared with AlexNet [23–24], GoogleNet [25–26], VGG [27–28], ResNet [29–30] and other networks, ZF has a low degree of structural complexity, and can also reduce the time complexity of feature extraction. The network structure of feature extraction is shown in Figure 4.

Figure 4.

The accuracy of the ship recognition is directly related to the quality of the candidate region which is taken in the target detection algorithm. The algorithm combines the candidate regions with CNN. If we can find a way to extract only a few hundred of high quality ship candidate regions and have a high recall, we can not only promote the effect of target recognition, but also upgrade the processing speed of ship recognition. Region Proposal Network (RPN) can be a good solution to the above problem. RPN uses information such as textures, edges, and colors in the image to find out where the ship may exist in image. It can ensure that a higher recall can be maintained when fewer regions (thousands or even hundreds) are selected. This greatly reduces the time complexity of subsequent operations, and the candidate regions obtained are of higher quality than the target of the traditional sliding window method.

The structure of regional proposal network is shown in Figure 5.

Figure 5.

A fixed-size window slide is used above the feature map extracted by the feature extraction network. The center point of each window corresponds to k anchor points, and each anchor point corresponds to multiple sizes and aspect ratios of sliding windows (see Fig. 5). On the right side, RPN uses 3 sizes and aspect ratios, so each slide window has k = 9 anchor points. Correspondingly, in the position of each sliding window, nine regions are simultaneously proposed, so the classification layer outputs 2×9 parameters that reflects the probability of the region where ship is. The bounding box regression layer has 4×9 parameters, representing the vertex coordinate of the nine proposal regions.

After using RPN, Faster R-CNN uses Fast RCNN to recognize and classify. Fast RCNN was proposed by Ross in 2015. This method solves the problems of RCNN detection slowly and training time consuming highly, and achieves end-to-end joint training. The RPN and Fast RCNN share the convolutional features in Faster R-CNN. Fast RCNN uses the high quality proposal regions provided by RPN, which greatly increases the speed of ship recognition. The schematic diagram of the Faster R-CNN is shown in Figure 6.

Figure 6.

This article uses the RPN and Fast RCNN two-part network for ship recognition. Therefore, the back propagation algorithm cannot be used directly during the training process. Therefore, alternate training methods are used during training.

In the first stage, train RPN. The ImageNet pre-trained model (M0) is used to initialize the RPN network, and then the dataset is used to train the RPN network. After the training is completed, the model M1 is obtained.

In the second stage, train Fast RCNN. The ImageNet pre-trained model (M0) is also used to initialize the Fast RCNN network. Then the trained RPN network in first stage is used to obtain the proposal area P1, and the P1 is used to train the Fast RCNN network. After the training is completed, the model M2 is obtained.

In the third stage, train the RPN network again. Using M2 to initialize the RPN network and get the M3 network, this stage only fine-tunes the parameters of the RPN and sets the network parameters of ZF.

In the fourth stage, Fast RCNN is trained again. Use the M3 network to initialize the Fast RCNN, then use the third stage of the trained RPN network to obtain the proposal regions P2, and use P2 to train the Fast RCNN. This stage only fine tunes the parameters of the full connecting layer of Fast RCNN. In this way, both networks share the convolutional layer and form a unified network. The number of iterations for each stage is shown in Table 1.

The algorithm adopted in this paper uses an end-to-end network for ship recognition, which avoids the trouble of buffering transfer data in the previous multi-stage training. It has greatly improved both in speed and accuracy, and achieved robust, rapid and accurate of ship recognition.

The dataset is provided by Data Fountain and is composed of large numbers of ships photoed in bad weather, including four classes of ship: cargo ship, cruise ship, fishing ship, yacht. In order to show the recognition results better in the experiments, we call these four kinds of ships as huochuan, youlun, yuchuan and youting. Parts of the sample image are shown in Figure 7:

Figure 7.

In the experiment, the dataset consisted of 33,397 images, of which the number of huochuan is 33756, the number of youlun is 8028, the number of youting is 13,608, and the number of yuchuan is 10,120. In addition, 5000 images with fog and coastal background were selected as negative samples of another two classes. As in deep learning, the background is also a class in the target recognition, so this experiment is set to 7 classes.

Since the recognition model adopted in this experiment is Faster R-CNN, the location of the ship must be marked in the training set firstly. That is, finding the region of interest (ROI), and setting the ground-truth of ships in the image, and mark the specific class of ship. The ROIs of some training samples are shown in Figure 8.

Figure 8.

In order to evaluate the effectiveness of the algorithm, this paper uses precision and recall to measure the performance of the model. Both the recall and the precision range are between [0,1]. The calculation equations are as shown in (5) and (6): (5) precision=TPTP+FP precision = {{TP} \over {TP + FP}} (6) recall=TPTP+FN recall = {{TP} \over {TP + FN}}

In the above formula, TP is the number of samples which is recognized correctly, FP is the number of false recognition, and FN is the number of missed recognition.

In order to verify the removal of fog algorithm used in this paper is practical and effective, some samples with fog are selected from the dataset for testing. The experiment results are shown in Figure 9.

Figure 9.

From the above experiment results, we can see that the method used in this paper not only can defog effectively, but also can retain the effective features of the ship as much as possible, and eliminate the unfavorable factors for ship recognition.

To further verify the efficiency of the proposed algorithm, the algorithm is compared with Faster R-CNN, 100 images (including 341 ships) are selected from the training set for testing. The precision and recall of the two algorithms are shown in the table 2.

From Table 2 above, it can be seen that the model trained by our method is higher than the model trained by Faster R-CNN in terms of precision or recall. Compared with Faster R-CNN, the precision is increased by 6.43%, the recall is increased by 4.68%. It can recognize the various classes of ship in the images. From Figure 10, it can be seen that both the precision and confidence of ship recognition have increased significantly.

Figure 10.

In order to verify the robustness of the proposed algorithm, large numbers of images that contain various ships under different sea states were selected from the dataset for testing. Its recognition result is shown in Figure 11.

Figure 11.

It can be seen from Figure 11, our method can deal with the interference of the coastal background as shown in (a), (d), (e), the fog cover as shown in (c), the size of the ship differs greatly as shown in (d), the small ships as shown in (e), and the various classes of ships are as shown in (f), all have better recognition results. The experiment results show that the proposed algorithm is widely applicable to the ship recognition under complex sea states, and the accuracy and real-time performance meet the actual requirements, and it has strong robustness.