Research on Image Super-resolution Reconstruction Based on Deep Learning

SRCNN [5] uses the relationship between deep learning and traditional sparse coding as a basis, and divides the 3-layer network into feature extraction, nonlinear mapping, and final reconstruction. For a low-resolution image, as shown in Figure 2, the method first uses bicubic interpolation to enlarge it to the same size as the target, and then extracts and represents the image block, and then uses a three-layer convolutional network to make nonlinear The result of mapping and reconstruction is output as a high-resolution image. SRCNN introduces convolutional neural networks to SR tasks for the first time. Unlike the step-by-step processing of traditional SR algorithms, SRCNN integrates various stages into a deep learning model, which greatly simplifies the SR workflow and can be regarded as a super deep learning based Milestones in resolution methods [6].

Figure 2.

Although the network structure of SRCNN is simple in design and is superior to traditional super-resolution algorithms in terms of image reconstruction quality and speed, it has the following problems: ①Does not use any prior knowledge; ②It is only suitable for SR tasks with a single magnification factor. For different magnification factors, the network model needs to be trained again; ③ Because the input image needs to be interpolated and magnified to the same size as the target, the entire image reconstruction process is performed in the HR space, which takes up a lot of memory space and increases the amount of calculation., The error produced by the interpolation process will also have an impact on the reconstruction effect, the model convergence speed is slower, and the training time is longer; ④The number of network layers is less, and the receptive field of the convolution kernel is also small (13 × 13). Good application of image context-related information, resulting in unclear texture of the final reconstructed HR image and limited algorithm adaptability [7].

Initially, the smaller datasets Set5 and Set14 were used to train the SRCNN algorithm. After training, the knowledge learned is relatively small, and the image reconstruction performance is constrained. When the relatively large dataset BSD200 is used, the reconstruction performance is significantly improved. The reconstruction performance of the image is also greatly affected by the size of the data set [8].

After that, although experts have proposed various network models, SRCNN is still used as a benchmark experiment for evaluating the performance of other network models.

In 2016, Shi et al. proposed an Efficient Sub-Pixel Convolutional Neural Network (ESPCN) network model based on pixel rearrangement [9], The core concept of ESPCN is a sub-pixel convolutional layer, which performs a convolution operation on the LR image to obtain LR image features, and then expands the features in the LR space to the HR space through the sub-pixel convolutional layer. Reorganize the HR feature map obtained after convolution to obtain an HR image [10].

The ESPCN network mainly improves the reconstruction layer of SRCNN. The LR image is used as the network input. The sub-pixel convolutional layer is used in the reconstruction layer to double the network training speed. The simple network structure and extremely high reconstruction speed make it very It is suitable for high-speed real-time systems that require relatively low reconstruction performance. In the ESPCN network, the interpolation function used for image size enlargement is implicitly included in the previous convolutional layer, which can be automatically learned. Since the convolution operation is performed on the size of the low-resolution image, the model efficiency is higher. The sub-pixel convolutional layer proposed by the ESPCN model has been widely used later. Compared with the deconvolutional layer proposed in the FSRCNN model, the learned nonlinear effect of upsampling from low-resolution images to high-resolution images is better. It is worth noting that the model also modified the activation function, replacing the ReLU function with the tanh function, and the loss function is the mean square error.

VDSR (Very Deep CNN for SR) [11] is the first deep model that proposes to use the global residual learning idea to solve the image SR problem. It is an improved network based on SRCNN. It uses a multi-layer convolution kernel for deep convolution, which not only reduces the amount of parameters, but also makes the following the network layer has a larger receptive field, can better utilize the image context information of a larger area, and obtain a better reconstruction effect than SRCNN. The biggest feature of VDSR is the deep network layers, good image reconstruction effect, and faster training speed. Since the author found that the input LR image is very similar to the output HR image, that is, the low frequency information carried by the LR image is very similar to the low frequency information of the HR image [12], So only need to learn the high-frequency residual part between the HR image and the LR image. The VDSR network structure is shown in Figure 3. It will interpolate to obtain an LR image of the same size as the target and input it into the network, and then add this image and the residual error learned by the network to obtain the final HR reconstructed image. The adaptive gradient pruning strategy is to train the network with a higher learning rate. Although the architecture is huge, it can still speed up the convergence. Therefore, on the basis of increasing the network depth, combining residual network and adaptive gradient cropping to accelerate model training can improve network performance and achieve better reconstruction effects. At the same time, through mixed training of images of different scales, the VDSR network can achieve a single Multi-scale SR reconstruction of the model.

Figure 3.

In 2018, Zhang et al. [13] It is believed that the input image contains a large amount of low-frequency information, and the existing SR network treats all channels of this information equally, and lacks the ability to distinguish and learn these channels, which hinders the network's characterization ability. Therefore, the deep residual channel attention network RCAN is proposed [13]. RCAN is the first network to apply the attention mechanism to SR problems. The algorithm obtains a weight value by learning the importance of different channels, which is equivalent to modeling the relationship between channel features, adaptively adjusting each channel feature, thereby effectively strengthening useful feature channels while suppressing useless feature channels, to make fuller use of computing resources. The model uses a locally nested residual structure (residual in residual), which is composed of a residual group (RG) and a long jump connection (LSC). A deeper network is built by simply stacking residual blocks and passes between feature channels the dependence relationship of the selection contains more key information feature channels, and enhances the identification and learning ability of the entire network.

The Generative Adversarial Network (GAN) was proposed by Goodfellow et al. It is inspired by the two-person zero-sum game in game theory. The two players in the GAN model are respectively composed of a generative model and a discriminant model (discriminative model) as [14]. SRGAN first applied adversarial training to the problem of image super-resolution reconstruction. The results show that the introduction of adversarial training can enable the network to generate finer texture details. GAN can complete many incredible generation problems, in the fields of image generation, speech conversion, and text generation. Occupy a very important position. As shown in Figure 4, SRGAN inputs the LR image to the generator G for image reconstruction. The discriminator D will train the generated image against the HR image, and finally output the image generated by the training [15]. The collaborative training of the generator and the discriminator enables the network to not only judge the similarity between the generated image and the actual high-resolution image in the pixel domain, but also pay more attention to its distribution similarity in the pixel space. Compared with the previous algorithm, although SRGAN is relatively low in objective evaluation indicators (such as PSNR), it has a better reconstruction effect in visual effects, image details and other intuitive aspects. This is related to its unique network structure and the loss function that combines perceptual loss and adversarial loss. Perceptual loss is a feature extracted by using convolutional neural networks. The generated image is compared with the target image. The feature difference after the convolutional neural network makes the generated picture and the target picture more similar in semantics and style. The confrontation loss is provided by GAN, and the network is trained according to whether the image can be successfully deceived. SRGAN is a milestone in the pursuit of the development of visual experience. SRGAN has significantly improved the overall visual quality of PSNR-based reconstruction.

Figure 4.

In order to improve the reconstruction accuracy of the SRGAN model, on this basis, ESRGAN [15] model has been improved in three aspects: the generator architecture is changed, the batch normalization (BN) layer is removed to reduce the artifacts generated in the reconstructed image, and a denser with higher reconstruction accuracy is introduced. Residual block (RDDB) to improve the structure of the model, make it more capacity and easier to train, help to improve generalization ability, reduce computational complexity and memory usage; improve the perceptual domain loss function, use the VGG before activation Features. This improvement will provide clear edges and more visually consistent results, which can better maintain image brightness consistency and restore better detail texture; enhance the discriminator's ability to discriminate, and use relative discriminators to make them then there is the absolute value of true and false but the relative distance between the predicted generated image and the real image.

Compared with the SRGAN model, on most standard test images, the ESRGAN model can reduce the noise in the reconstructed image while maintaining better details. A large number of experiments have shown that enhanced SRGAN, called ESRGAN, is always better than the most advanced methods in sharpness and detail.

近年来，深度学习技术蓬勃发展，基于深度学习的超分辨率重建方法已逐步成为图像超分 辨率的主流，并发展出了一系列以 CNN 和 GAN 为基础的网络模型，2.1–2.2 节分别介绍了基于 CNN 网络模型和基于 GAN 网络模型的超分方 法。整体来看，算法的重建性能在不断提升， 重建后的图像与原始图像越来越接近。基于 GAN 网络模型的算法重建得到的图像纹理细节 较优，提升了图像的真实感，人眼感知效果较 好。与之相比，基于 CNN 网络模型的算法网络 复杂度更低，训练难度更小，重建结果精度更 高，但会产生伪影且重建速度更慢。

像素损失主要是度量预测图像和目标图像之 间的像素差异，主要包括了 L₁（平均绝对误差） 损失和 L₂（均方误差）损失。

L₁ 损失，也被称为最小绝对偏差（LAD）和 最小绝对误差（LAE），计算的是实际值与目 标值之间绝对差值的总和。在有监督的图像超 分辨率任务中，目标是使得生成的图像（SR） 尽可能接近真实的高分辨率图像（HR），使用 L₁ 损失计算的则是 SR 和 HR 对应像素位置的值之间的误差。平均绝对值误差（MAE）的基本 表达式如式（1）所示： (1) LMAE(θ)=1n∑i=1n|Yi−F(Xi, θ)| {L_{MAE}}\left( \theta \right) = {1 \over n}\sum\limits_{i = 1}^n {\left| {{Y_i} - F\left( {{X_i},\,\theta } \right)} \right|}

其中，L(θ)表示网络需要优化的损失函数，n 表示训练样本的数目，θ 表示深度神经网络的参 数，F(X_i, θ)为网络重建后的图像，Y_i 是表示对 应的 HR 图像。L₂损失，也叫均方误差损失 （MSE），计算的是实际值与目标值之间绝对 差值的平方总和，使得基于深度学习的图像超 分辨率模型在性能方面有了很大的提升，是最 常用的损失函数。但是 MSE 损失能够对大的损 失进行惩罚，对于小的损失上却无所作为，因 此会产生模糊的图像。其基本表达式如式（2） 所示： (2) LMSE(θ)=1n∑i=1n‖Yi−F(Xi, θ)‖2 {L_{MSE}}\left( \theta \right) = {1 \over n}\sum\limits_{i = 1}^n {{{\left\| {{Y_i} - F\left( {{X_i},\,\theta } \right)} \right\|}^2}}

其基本符号意义与式（１）中一致。最小均 方误差的应用有效解决了 SR 重建图像与目标 HR 图像之间差值衡量问题，使得基于深度学习 的图像 SR 模型相对传统基于学习的 SR 重建模 型有了较大的提高。

与 MSE 相比，MAE 有个优点，即它对离群 点有更好的鲁棒性，更具有包容性。由式（1） 可知，MAE 计算的是误差 Yi-F(Xi, θ)的绝对值， 所以无论是 Yi-F(Xi, θ)>1 还是 Yi-F(Xi, θ)<1， 没有平方项的作用，惩罚力度都是相同的，所 占权重相同。

由于这些损失函数分别对每个像素向量的类 预测进行评估，然后对所有像素进行平均，因 此它们断言图像中的每个像素都具有相同的学 习能力。

与 L₁ 损失函数相比，L₂ 损失函数会放大最大 误差和最小误差之间的差距（比如 2*2 和 0.1*0.1），另外 L₂ 损失函数对异常点也比较敏 感。

由于峰值信噪比（PSNR）的定义（见 4.1） 与像素差相关度较高，且最小化像素损失等同 于将 PSNR 最大化，所以像素损失函数逐渐成 为被普遍使用的损失函数。但由于其并不探究 图像质量问题(例如，感知质量^[16]，纹理细节^[17])，因此结果往往欠缺高频细节内容，产生过于平 滑的纹理细节。

对于图像风格化，图像超分辨率重建等任务来说，早期都使用了图像像素空间的 L₂ 损失， 但是 L₂ 损失与人眼感知损失的图像质量并不契 合，恢复出来的图像细节往往表现较差。如今 的研究中，L₂ 损失逐步被人眼感知损失所替代。 人眼感知损失也被称为感知损失（perceptual loss），其与 MSE 损失采用图像像素进行求差 的不同之处在于所计算的空间不再是图像空间。 基于感知的损失函数可以恢复更多的高频信息， 使重建性能更好。目前，基于感知的损失函数 主要有内容损失和对抗损失。

内容损失函数又分为特征重建损失函数和风 格重建损失函数。Bruna 等人^[18]首次提出特征重 建损失函数，通过预训练的 VGG19 网络分别提 取重建 SR 图像与 HR 图像在特征空间中对应的 特征映射且进行比较，其表达式如下： (3) LVGG/j=1WjHj∑h=1Hj∑w=1Wj(ϕj(Y)x,y−ϕj(GθG(X))x,y)2 {L_{VGG/j}} = {1 \over {{W_j}{H_j}}}\sum\limits_{h = 1}^{{H_j}} {\sum\limits_{w = 1}^{{W_j}} {{{\left( {{\phi _j}{{\left( Y \right)}_{x,y}} - {\phi _j}{{\left( {{G_{{\theta _G}}}\left( X \right)} \right)}_{x,y}}} \right)}^2}} }

其中，W_j 和 H_j 表示第 j 幅特征图的宽与高， ф_j 表示 VGG19 网络内第 j 次卷积获得的特征映 射，X 表示原始 LR 图像，Y 为重建后的 HR 图像，G_θG 表示网络生成的 SR 图像。Gatys 等人^[19] 为了使重建 SR 图像与 HR 图像的纹理细节等保 持一致，在特征重建损失函数的基础上又提出 了风格重建损失函数，该函数定义了一个 Gram 矩阵，计算公式如下： (4) Gjϕ(X)c,c′=1CjWjHj∑h=1Hj∑w=1Wjϕj(X)h,w,c ϕj(X)h,w,c′ G_j^\phi {\left( X \right)_{c,c'}} = {1 \over {{C_j}{W_j}{H_j}}}\sum\limits_{h = 1}^{{H_j}} {\sum\limits_{w = 1}^{{W_j}} {{\phi _j}{{\left( X \right)}_{h,w,c}}{\phi _j}{{\left( X \right)}_{h,w,c'}}} }

经过 VGG 网 络 提 取 到 的 特 征 图 大 小 为 C_j×W_j×H_j。随后在对应层中计算 Gram 矩阵的欧 式距离差并相加得到风格重建损失函数，如下 式： (5) Lstyle/jϕ=‖Gjϕ(Y)−Gjϕ(GθG(X))‖2 L_{style/j}^\phi = {\left\| {G_j^\phi \left( Y \right) - G_j^\phi \left( {{G_{{\theta _G}}}\left( X \right)} \right)} \right\|^2}

SRGAN 网络首次提出对抗损失，其基本形式 如下： (6) LGenSR=∑n=1N−1bDθD(GθG(X)) L_{Gen}^{SR} = \sum\limits_{n = 1}^N { - {\it 1}b{D_{{\theta _{\rm{D}}}}}} \left( {{G_{{\theta _G}}}\left( X \right)} \right)

其中，Ｎ表示图像数量，θ_Ｄ表示鉴别网络的 参数，θ_Ｇ表示生成网络的参数，表示生成图像， 是真实的 HR 图像的概率。网络最终的优化目标 为一个最小最大化问题： (7) minθGmaxθD EIHR□Ptrain(IHR)[1bDθD(IHR)]+EIHR□PG(IHR)[1−1bDθD(GθG(ILR))] \mathop {\min}\limits_{{\theta _G}} \mathop {\max}\limits_{{\theta _D}} \,{E_{{I^{HR}}\square {P_{train\left( {{I^{HR}}} \right)}}}}\left[ {{\it 1}b{D_{{\theta _D}}}\left( {{I^{HR}}} \right)} \right] + {E_{{I^{HR}}\square {P_{G\left( {{I^{HR}}} \right)}}}}\left[ {1 - {\it 1}b{D_{{\theta _D}}}\left( {{G_{{\theta _G}}}\left( {{I^{LR}}} \right)} \right)} \right]

其中，Ｐ_train（I^HR）表示 HR 图像分布，Ｐ_Ｇ （I^LR）表示原始 LR 图像分布，对抗训练使得 生成的 SR 图像与原始的 HR 图像高度相似，使 得判别网络难以辨别，最终得到能够以假乱真 的 SR 图像。SRGAN 的判别目标为输入图像是 否为真，与 SRGAN 不同，ESRGAN 用相对平 均判别器替代了判别网络的判别器，判别目标 为预测真实 HR 图像比生成 SR 图像更真实的概 率。判别网络如式（8）和式（9）所示： (8) D(xr)=σ(C(real))→1D(xf)=σ(C(fake))→0 \matrix{ {D\left( {{x_r}} \right)} & = & {\sigma \left( {C\left( {real} \right)} \right) \to 1} \cr {D\left( {{x_f}} \right)} & = & {\sigma \left( {C\left( {fake} \right)} \right) \to 0} \cr } (9) DRa(xr, xf)=σ(C(real)−E[C(fake)])→1DRa(xf, xr)=σ(C(fake)−E[C(real)])→0 \matrix{ {{D_{Ra}}\left( {{x_r},\,{x_f}} \right)} & = & {\sigma \left( {C\left( {real} \right) - E\left[ {C\left( {fake} \right)} \right]} \right) \to 1} \cr {{D_{Ra}}\left( {{x_f},\,{x_r}} \right)} & = & {\sigma \left( {C\left( {fake} \right) - E\left[ {C\left( {real} \right)} \right]} \right) \to 0} \cr }

其中，real 表示真实 HR 图像，fake 表示生成 SR 图像，C(real)表示鉴别器判断结果，E[C(fake)] 表示鉴别网络判断结果的平均值。其中 σ 表示 Sigmoid 激活函数，通过改进判别器帮助网络学 习更敏锐的边缘和更多的纹理细节。