Super-resolution Image Reconstruction Based on Double Regression Network Model

Attention mechanisms stem from studies of the human visual system. Human vision tends to focus on salient areas and ignore useless information, which increases the efficiency with which the brain processes information. The compression-expansion module proposed by Hu et al. using the attention mechanism can explicitly model the interdependence between feature channels, automatically obtain the importance of each feature channel through learning, and then suppress the less useful features according to the importance, so as to improve the performance of image classification network. The RCAN method proposed by Zhang et al. introduces the attention model of Hu et al. The channel attention module can adaptively select feature channels with richer information, which improves the performance of SISR. However, the feature channels of this attention mechanism are independent of each other, which limits the flow of feature information between channels.

Increasing the network depth can generally improve the performance of SISR. However, deeper networks, which generally have more parameters, require more training data. In practice, the acquisition of training data is often limited, and the risk of overfitting of network training is also increasing. One advantage of a recursive network is that it can increase the depth of the network without increasing the number of parameters. DRCN introduces recursion techniques into SISR methods for the first time: a convolutional layer is used as a recursion unit, and the weights are shared among the recursion units. DRRN improves the use of recursive techniques: residual blocks are used as recursive units, and parameters are shared among residual blocks, which also improves the performance of DRCN. Similarly, MemNet, proposed by Tai et al., also uses residual blocks as recursive units to build the network.

The image super-resolution under the dual regression model focuses on the input LR image to achieve the purpose of restoring HR image. In principle, LR image I_xLR Can be expressed as the output of HR image after degradation function. (1) IxLR=d(IyHR,∂) {I_{xLR}} = d\left( {{I_{yHR}},\partial } \right)

Where d is the SR degradation function responsible for converting HR image into LR image, I_yHR Is the input HR image, which is the reference image, and the input parameters that describe the degradation function of the image. The degradation parameters are usually the scaling coefficient, blur type, and noise. In practice, the degradation process and the dependent parameters are unknown and usually only LR images are used to obtain HR images via SR methods. The SR process is responsible for predicting the inverse of the degradation function d. (2) g(IxLR,δ)=d−1(IxLR)=IyE≈IyHR g\left( {{I_{xLR}},\delta } \right) = {d^{ - 1}}\left( {{I_{xLR}}} \right) = {I_{yE}} \approx {I_{yHR}}

Where g is the SR function and describes the input parameters of the function g, I_yE is corresponds to the input I_xLR of the estimated HR image. It is also worth noting that the super-resolution function in Eq. (2) is ill-posed because the function g is a non-mapping function. As a result, I_yE not the only one. There are many possibilities. The degradation process of the input LR image is unknown, and this process is affected by many factors such as sensor-induced noise, artifacts due to lossy compression, speckle noise, motion blur, and misfocused images. In most studies, a single down sampling function is used as the image degradation function. (3) d(IyHR,∂)=(IyHR)↓Sf, {s}⊆∂ d\left( {{I_{yHR}},\partial } \right) = \left( {{I_{yHR}}} \right){ \downarrow _{{S_f}}},\,\left\{ s \right\} \subseteq \partial

Thereinto, →_Sf represents the down-sampling operation, S_f Is the sampling factor. One of the down-sampling functions often used in SR is bicubic interpolation with antialiasing. In order to simulate a more realistic environment, researchers use more operations in the down-sampling function, and the overall down-sampling operation is: (4) d(IyHR,∂)=(IyHR⊗κ)↓Sf+nσ, {κ, s, σ}⊆∂ d\left( {{I_{yHR}},\partial } \right) = \left( {{I_{yHR}} \otimes \kappa } \right){ \downarrow _{{S_f}}} + {n_\sigma },\,\left\{ {\kappa ,\,s,\,\sigma } \right\} \subseteq \partial

Where I_yHR ⊗ κ represents HR image, I_yHR convolution κ with fuzzy kernel, n_σ Is additive white Gaussian noise with σ standard deviation. The degradation function defined in Eq. (4) is closer to the actual function because it considers more parameters than the simple down-sampled degradation function. The purpose of image super resolution is to minimize the loss function, as follows: (5) ϕ^=[min L(IyE, IyHR)]ϕ+hψ(ϕ) \hat \phi = {\left[ {\min \,L\left( {{I_{yE}},\,{I_{yHR}}} \right)} \right]_\phi } + h\psi \left( \phi \right)

Where L(I_yE, I_yHR) is the loss function between the output HR image after SR and the actual HR image, ψ (ϕ) Is the regularization term. The most commonly used loss function in SR is based on the pixel mean square error, which can also be called the pixel loss. Figure 2 is a schematic diagram of the super-resolution reconstruction process.

Figure 2.

One of the learning strategies based on the basic network is to recursively learn high-level features using the same module. This approach also minimizes model parameters, since the strategy requires only one module to be recursively updated, as shown in Figure 3.

Figure 3.

One of the most commonly used recursive networks is the deep recursive convolutional network. With one convolutional layer, the DRCN can reach a receptive field of 41×41 without additional parameters. The deep recursive residual network uses the residual module Res Block as a part of the recursive module, which has a total of 25 recursions and performs better than the baseline Res Block. In addition to end-to-end recursion, the researchers also used a two-state recursive network, which shares signals between LR images and generated HR image states within the network. In general, while reducing parameters, recursive learning networks can learn complex representations of data at the cost of computational performance. In addition, increased computational requirements may cause gradients to explode or disappear. Therefore, recursive learning is often used in conjunction with multi-supervised or residual learning to minimize the risk of gradient explosion or disappearance.

Existing methods only focus on learning the mapping from LR to HR images. However, the space of possible mapping functions can be very large, making training very difficult. To solve this problem, we propose a dual regression scheme by introducing an additional constraint on LR data. Specifically, in addition to learning the LR→HR mapping, we also learn the inverse/dual mapping from the super resolution image to the LR image. We learn to reconstruct HR images with the original map P and LR images with the dual map D simultaneously. Note that the dual map can be viewed as an estimate of the underlying down-sampled kernel. Formally, we formulate the SR problem as a binary regression scheme involving two regression tasks. The specific expression form is shown in Figure 1.

The original regression task: We need to find a function P: Y → Y Make the prediction P (x) similar to its corresponding HR image Y.

Dual regression task: We seek a function D: Y → X, so that the prediction of D(y) is similar to the original input LR image x.

Primal and dual learning tasks can form a closed loop to provide information supervision for training models P and D. If P (x) is the correct HR image, then the down-sampled image D(P (x)) should be very close to the input LR image x. With this constraint, we can reduce the space of functions of possible maps, making it easier to learn better maps to reconstruct HR images. By jointly learning these two learning tasks, we propose to train the super-resolution model as follows. Given N pairs of samples Sp={(xi,yi)}i=1N {S_p} = \left\{ {\left( {{x_i},{y_i}} \right)} \right\}_{i = 1}^N Where x_i and y_i denotes those in the paired data set i for low resolution and high resolution images. The training loss can be written as (6) ∑i=1NζP(P(xi), yi)+λζD(D(P(xi)),xi) \sum\limits_{i = 1}^N {{\zeta _P}\left( {P\left( {{x_i}} \right),\,{y_i}} \right) + \lambda {\zeta _D}\left( {D\left( {P\left( {{x_i}} \right)} \right),{x_i}} \right)}

Where ζ_P and ζ_D represent the loss functions of primal and dual regression tasks, respectively. Here, λ controls the weight of the dual regression loss.

We consider a more general SR case where there is no corresponding HR data with real-world LR data. More critically, the degradation methods of LR images are often unknown, which makes this problem very challenging. In this case, existing SR models often produce serious adaptation problems. To solve this problem, we propose an efficient algorithm to adapt the SR model to the new LR data. Dual regression maps learn the underlying degradation methods and do not necessarily depend on HR images. Therefore, we can directly learn from unpaired real-world LR data for model adaptation. In order to ensure the performance of HR image reconstruction, we also add the information of pairwise synthetic data which is very easy to obtain. Given M unpaired LR samples and N paired synthetic samples, the objective function can be expressed as follows. (7) ∑i=1M+NιSp(xi)ζD(P(xi)),yi)+λζD(D(P(xi)),xi) \sum\limits_{i = 1}^{M + N} {{\iota _{{S_p}}}\left( {{x_i}} \right){\zeta _D}\left. {\left( {P\left( {{x_i}} \right)} \right),{y_i}} \right) + \lambda {\zeta _D}\left( {D\left( {P\left( {{x_i}} \right)} \right),{x_i}} \right)}

Where, ι_Sp (x_i) is an index function, if x_i ∈ S_p, the function is equal to 1, otherwise the function is equal to 0.