Development of Blind Deblurring Based on Deep Learning

In order to limit the solution space to effective natural images, the existing restoration technologies [6, 7, 8] explicitly use the image priors made by hand through empirical observation. However, designing such a priori knowledge is a challenging task and often can not be popularized. In order to improve this problem, deep learning methods begin to be more applied to image processing. Recently, the most advanced method [17, 44,] adopts convolution neural network (CNN). After 2016, the early CNN based image deblurring method usually uses CNN as a fuzzy kernel estimator to construct a two-stage image deblurring framework, such as CNN based fuzzy kernel estimation stage and kernel based deconvolution stage. Chakrabarti uses CNN to estimate the fuzzy kernel [12], obtains the fuzzy kernel, and then uses the non blind deblurring algorithm to deblurring. Schuler et al. [13] trained a depth network to estimate the fuzzy kernel, and then used the traditional non blind deconvolution method to restore the potentially clear image. Sun et al. [14] used convolutional neural network to estimate the image fuzzy kernel, and then used the estimated fuzzy check image for deblurring restoration. Although this method uses the blind deblurring algorithm to restore the image, it still uses the idea of non blind deblurring after estimating the fuzzy kernel to deconvolute the image.

This leads to the slow operation of the algorithm and the restoration result depends on the fuzzy kernel estimation. However, this method applies the convolution neural network to image deblurring, which lays the foundation for subsequent methods based on this. Li et al. [15] proposed a new convolution structure called “hole convolution”, and its kernel is calculated by a rectangular rectangular ring, The experimental results show that this method can effectively restore the image; Liu et al. [16] proposed a two-stage deblurring module to restore the blurred image of the dynamic scene based on the high-frequency image. Firstly, the residual image is thinned by the coding network, and then the thinned residual image is combined with the input blurred image to obtain the latent image, and further proposed a coarse to fine framework based on the fuzzy processing module. Noroozi uses multi-scale CNN for end-to-end training [17], which does not need to estimate the fuzzy kernel and belongs to blind fuzzy. The basic idea of the blind deblurring method based on CNN is to take the blurred image and the corresponding clear image as training samples and input them into the convolution neural network for training. After the training, the optimized network model is obtained. When in use, the blurred image is taken as the network input, and the network output is the deblurring image. Different image processing is based on different convolution networks, as shown in Fig. 2.

Figure 2

On the other hand, the recent image deblurring method based on CNN aims to directly understand the complex relationship between blurred and clear image pairs in an end-to-end way. NAH et al. [18] proposed using multi-scale convolutional neural network to defuzzify and created the most widely used GoPro data set at present. The “end-to-end” training method is adopted, which has good model effect and operation speed. It can directly recover the latent image without assuming any limited fuzzy kernel model. In particular, the multi-scale structure is designed to imitate the traditional coarse to fine optimization method. Unlike other methods, this method does not estimate explicit potential errors. Therefore, artifacts due to kernel estimation errors will not be generated. Secondly, the proposed model is trained by multi-scale loss. As shown in the figure, the model is suitable for the structure from coarse to fine, which greatly enhances the convergence. The multi-scale system uses the improved residual network structure, as shown in Figure 3, to achieve a deeper architecture. In addition, the results are further improved by using counter loss [19]. Because the loss term optimizes the result and makes it similar to the ground truth, it even restores the extremely complex occlusion area of the fuzzy kernel, and has made significant improvements in the deblurring of dynamic scenes. A large number of experimental results show that the performance of this method in qualitative and quantitative evaluation is much better than the latest dynamic scene deblurring method.

Figure 3

The cyclic neural network is used in the image motion fuzzy restoration method. This method innovatively uses the cyclic neural network processing data with sequence and time dependence for image processing.

Tao et al. [20] proposed a scale cyclic neural network (SRN deblurnet) according to the strategy of gradually recovering clear images with different resolutions in the pyramid, Compared with other methods, SRN has simpler structure, fewer parameters and easier training than other networks. Resblock network is also inspired by the recent success of encoder decoder structure used in various computer vision tasks, and explores an effective method to adapt it to image deblurring tasks. In SRN network, the direct application of the existing encoder decoder structure can not produce the best results. The encoder decoder resblock network of Tao et al. Amplifies the advantages of various CNN structures, produces the feasibility of training and produces a very large receptive field, which is very important for large motion deblurring. Experiments show that the end-to-end depth image deblurring framework can greatly improve the training efficiency by using the cyclic structure and combining the above advantages.

Zhang et al. [21] proposed a depth hierarchical multi patch network based on spatial pyramid matching, which processes fuzzy images through a fine to rough hierarchical representation, It runs 40 times faster than the previous multi-scale method. Zhang et al. [22] proposed a spatial variation neural network composed of three deep convolution neural networks (CNNs) and a cyclic neural network (RNN). RNN is used as a deconvolution operator to deconvolute the feature map extracted from the input image by a neural network. This method has good performance, speed and model size.

Most of the traditional deblurring methods based on convolutional neural network have a series of problems, such as the color of the output image is unnatural, the texture features are not rich enough, the image is too smooth and so on. Confrontation network (GAN) is also gradually applied to the field of image deblurring because it can retain texture details and generate realistic images. In 2014, goodflow et al. Proposed a groundbreaking generation confrontation network (GAN) that can show strong ability in computer vision tasks [23].

The images processed by the generation confrontation network are very close to clear images, It can not even be distinguished with the naked eye [24–25]. However, the generator should be constrained when using the generated countermeasure network, because once the network is too free, it will lead to instability, and it is difficult to learn the mapping relationship between the input image and the target image. If L1 or L2 constraints are directly established between the output of the generator and the corresponding target image at the pixel level, the generated image will become too smooth and vulnerable to image noise. With the increasingly prominent application of generative countermeasure network in the image field, ledig et al. [26] proposed a generative countermeasure network (GAN) for image super-resolution (SR). Its deep residual network can restore realistic texture, and the results are obtained through the mean score (MOS) test. Using srgan can improve the quality of perception, this method can be well applied to the field of image deblurring.

In 2018, kupyn et al. [27] first removed the blur of camera jitter according to the conditional counter-measure network, and then proposed a kernel free blind motion defuzzification learning method to make up for the previous shortcomings. The conditional countermeasure network deblurgan optimized by using multi-component loss function, but this method did not consider the impact of different feature layers in the perception network on the perception loss, The detail of the restored image is still smooth. Meanwhile, in 2019, kupyn et al. [28] introduced the feature pyramid network into deblurgan for the first time and proposed a new end-to-end generation countermeasure network deblurgan-v2 for single image motion deblurgan, which greatly improved the efficiency, quality and flexibility of deblurgan. Table 2 shows more depth learning methods for blind image deblurring. Then the network is reproduced, and the deblurring image is shown in Figure 4.

Figure 4

Image recognition is actually a classification process. In order to identify the category of an image, we need to distinguish it from other different categories of images. This requires that the selected features not only can well describe the image, but also can well distinguish different types of images. We want to select the image features with small difference between similar images (small intra class spacing) and large difference between images of different categories (large class spacing), which we call the most discriminative feature. In addition, prior knowledge plays an important role in feature extraction. How to rely on prior knowledge to help us select features is also a problem that will continue to be concerned later. The research process and ideas of traditional feature extraction methods are very useful, Because these methods have strong interpretability, they provide inspiration and analogy for designing machine learning methods to solve such problems. The existing convolutional neural network is similar to these feature extraction methods, because each filter weight is actually a linear recognition pattern, which is similar to the boundary and gradient detection of these feature extraction processes. At the same time, the role of pooling is to coordinate the information of a region, which is similar to the feature integration (such as histogram) after these features are extracted. Through experiments, it is found that the first few layers of convolution network are actually doing edge and gradient detection. However, in fact, when the convolution network was invented, there were no such feature extraction methods.

The evaluation of image quality can be divided into subjective evaluation and objective evaluation. Subjective evaluation mainly evaluates human visual senses, while objective evaluation uses an evaluation standard to compare image quality. The commonly used objective evaluation methods are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) [29]. The PSNR reflects the distortion degree of the estimated image and the original clear image. Generally speaking, the larger the peak signal-to-noise ratio, the better the image restoration effect. Its expression is: (2) PSNR=10log10[M×N×2552‖X−X^‖22] {\rm{PSNR}} = 10\log 10\left[ {{{{\rm{M}} \times {\rm{N}} \times {{255}^2}} \over {\left\| {{\rm{X}} - {\rm{\hat X}}} \right\|_2^2}}} \right]

M and N represent the size of the row and column of the image respectively, and X represents the original clear image, ◯ represents the estimated image. Due to research findings, PSNR is sometimes inconsistent with human visual evaluation. Therefore, SSIM is adopted to further improve the evaluation criteria. Its expression is: (3) SSIM=[l(x,y)]α[c(x,y)]β[s(x,y)]γ {\rm{SSIM}} = {\left[ {{\rm{l}}\left( {{\rm{x}},{\rm{y}}} \right)} \right]^\alpha }{\left[ {{\rm{c}}\left( {{\rm{x}},{\rm{y}}} \right)} \right]^\beta }{\left[ {{\rm{s}}\left( {{\rm{x}},{\rm{y}}} \right)} \right]^\gamma } (4) l(x,y)=2uxuy+c1ux2+uy2+c1 {\rm{l}}\left( {{\rm{x}},{\rm{y}}} \right) = {{2{{\rm{u}}_{\rm{x}}}{{\rm{u}}_{\rm{y}}} + {{\rm{c}}_1}} \over {{\rm{u}}_{\rm{x}}^2 + {\rm{u}}_{\rm{y}}^2 + {{\rm{c}}_1}}} (5) c(x,y)=2σxσy+c2σx2+σy2+c2 {\rm{c}}\left( {{\rm{x}},{\rm{y}}} \right) = {{2{\sigma _{\rm{x}}}{\sigma _{\rm{y}}} + {{\rm{c}}_2}} \over {\sigma _{\rm{x}}^2 + \sigma _{\rm{y}}^2 + {{\rm{c}}_2}}} (6) s(x,y)=σxy+c3σxσy+c3 {\rm{s}}\left( {{\rm{x}},{\rm{y}}} \right) = {{{\sigma _{{\rm{xy}}}} + {{\rm{c}}_3}} \over {{\sigma _{\rm{x}}}{\sigma _{\rm{y}}} + {{\rm{c}}_3}}}

SSIM is a comprehensive map image evaluation index, which evaluates the image from brightness, contrast and structural similarity respectively, where u_x and u_y represent the mean value of image X and Y respectively, X and Y represent the difference between image X and Y respectively, and XY represents the co difference between image x and y. SSIM measures the similarity of two images, and its value is between 0 and 1. The closer it is to 1, the higher the similarity is the better the restoration result is.

Most of the blurred images in the traditional data set are blurred by some fixed cores, which is difficult to imitate the natural blurred images. When we use the algorithm in machine learning to deal with some problems, the quality of the data set will directly affect the results of our algorithm. Therefore, high-quality data sets play an important role in our follow-up research. The most direct way to obtain the data set is to directly capture the image in the real scene, the data set of Lai et al. [30]. The image obtained in this way only has fuzzy image and no corresponding clear image, so it can only be used for testing and can not be used for the data set of network training. The restoration of fuzzy images through neural networks requires pairs of fuzzy clear image pairs for network training, but such data sets are difficult to obtain in the real world. The manual use of paired data sets can not ensure the consistent content of clear images and fuzzy images, and the fuzzy images synthesized by the algorithm can not contain various complex factors in the real environment. It may perform well when used for network training, but the effect is really unsatisfactory when using real fuzzy images.

In deep learning, the quality of data sets directly affects the subsequent experimental results. The quality and updating of data sets are of great significance to image deblurring. Among them, only 2103 pairs of training pictures and 1111 pairs of test pictures are used in GoPro dataset, which is the most widely used and the largest.

Compared with datasets in other fields of computer vision, especially the Imagenet dataset contains 14197122 pictures, it is very different. GoPro data sets only expand the number of data sets, the diversity of data sets obtained is not enough, and the scene is too single, and some even show that motion blur is not particularly obvious.

Similarly, some other data sets synthesized by algorithms show less data and are not sufficient in demonstration. Therefore, in the current situation, we need to enrich and update the data, not only to ensure that the amount of data is sufficient, but also to fully meet the requirements of the experiment. Different from image recognition or image segmentation, it is difficult to obtain fuzzy image data set. However, for any field, data sets are the basis of researchers' development. The lack of data sets directly affects the research progress in this field. Therefore, it is urgent to propose a large-scale and new data set.

The important factor affecting the application of the algorithm is the running speed of the deblurring algorithm. Some scenes require high real-time performance, and the efficiency of the algorithm is the first choice. To improve the timeliness of the image motion blur restoration algorithm, this algorithm can be applied to many scenes to improve the solution based on computer vision. For example, in factory production monitoring, more and more attention is paid to the use of image processing technology. In the traditional method, the articles need to stop at the monitoring point to collect images in the production process. Using the image motion deblurring algorithm with high real-time performance can collect pictures when the articles are moving, save the steps of stopping the articles, and greatly improve the production efficiency of the article production line. Therefore, improving the efficiency of the algorithm plays an important role in daily life.

Nowadays, people's subjective feeling is very close to the widely used structural similarity index and peak signal-to-noise ratio, especially when there is uneven fuzzy motion blur in the training image, so a reference evaluation index is good enough to evaluate the processed deblurring image and the rationality of the algorithm when processing the experimental results. With the continuous development of deblurring technology, in order to achieve a fair and fair evaluation of the deblurring image, we not only attach people's perception test effect on the image at the back of the paper, but also need to get a more recognized evaluation standard in the field of deblurring.