Adaptively Truncating Gradient for Image Quality Assessment

The image information is presented by the change in the intensity values in the spatial domain, and this change may be destroyed by degradation of the image quality. The gradient feature can effectively measure the change and is widely used in IQA algorithms. The image gradient can be obtained by convolving the image with a gradient operator, such as Sobel, Roberts and Scharr and Prewitt. Usually, a different gradient operator for the IQA model may yield distinguished performance. This problem was discussed in [2,6], where the experiment results showed that the Scharr operator can obtain a slightly better performance than the others. Here, we adopt a 3×3 Scharr operator whose templates along the horizontal (H) and vertical (V) directions take the following form:hH=116[30−3100−1030−3],hV=116[3103000−3−10−3]

Denote r = [r₁, ⋯, r₁, ⋯, r_N] for a reference image and d = [d₁,⋯,d_i,⋯,d_N] for a distorted image, where i is the pixel index, and N is the number of total pixels. The image gradients in the horizontal and vertical directions can be obtained by convolution of the image with h_H and h_V, and the gradient magnitude is computed from their root mean square. The gradient magnitudes of r and d at each pixel i, denoted as G(r, i) and G(d, i) are calculated as(1)G(r,i)=(r⊗hH)2(i)+(r⊗hV)2(i)(2)G(d,i)=(d⊗hH)2(i)+(d⊗hV)2(i)

Where the symbol ⊗ denotes the convolution operation.

The image gradient only reflects the objective changes in images. Since human evaluation of image quality is carried out in the HVS perception space, the image features extracted for IQA models should reflect the subjective changes perceived by the HVS. We consider that the ability of HVS to perceive changes is subject to the upper threshold. When the objective change exceeds the upper threshold, the subjective change does not obviously increase. In this study, we define the adaptively truncating gradient to measure the subjective change sensed by the HVS.

Denote T as the upper threshold. We define a truncating function trunc(·). For any given variable x, it is retained when it is less than T and truncated when it is greater than T. The specific expression is(3)trunc⁡(x)={T, if x≥Tx, if x<T

The truncating gradients of r and d at each pixel i are denoted as G_T(r, i) and G_T(d, i), and the upper threshold at this point is denoted as T(i). Using formula (3), the calculation of G_T(r, i) is as follows:(4)GT(r,i)=trunc⁡(G(r,i))={T(i), if G(r,i)≥T(i)G(r,i), if G(r,i)<T(i)

In Eq. (4), if the value of G(r, i) is greater than T(i), then G(r, i) will be truncated, and the truncating gradient G_T(r, i) is set to T(i). That is, the part of the gradient magnitude that is greater than the upper threshold is masked. Otherwise, G(r, i) is not be masked, and the truncating gradient G_T(r, i) is set equal to G(r, i). That is, the part of the gradient magnitude that is less than the upper threshold can be perceived by the HVS.

Similarly, using formula (3), G(r, i), is calculated as follows:(5)GT(d,i)=trunc⁡(G(d,i))={T(i), if G(d,i)≥T(i)G(d,i), if G(d,i)<T(i)

Obviously, for the calculation of the truncating gradients G_T(r, i) and G_T(d, i) Eq. (4) and (5), the selection of the upper threshold T(i) is very important. According to Weber’s law, the ratio of the stimulus change that causes a just noticeable difference (JND) from the original stimulus intensity is a constant. In psychology, the HVS has the property of light adaptation, and the perception of luminance obeys Weber’s law [7]. The just noticeable incremental luminance over the background by the HVS is related to the background luminance.

Inspired by this recognition, in contrast to Weber’s law, we consider that the upper threshold for truncating the significantly perceptible stimulus change is also related to the original stimulus intensity value. Because different pixels in the image correspond to different gray values, the original stimulus intensity values will also be different. Here, we adaptively determine the upper threshold according to the background luminance of different areas of the image.

The adaptively upper threshold is defined as(6)T(i)=I(i)T0

Where T₀ is an adjustable threshold parameter. (The details of selecting T₀ will be presented in section III-A.) I(i) takes the larger value of the luminance of r and d at point i.(7)I(i)=max(r¯(i),d¯(i))

In formula (7), the luminance values r¯(i) and d¯(i) at pixel i of r and d is estimated by formulas (8) and (9). For reference image r, denote the square neighborhood as Ωir with center of pixel i and radius of t, and let the intensity value of any pixel in the neighborhood be r_i,j, j∈Ωir. Similarly, for the distorted image, denote the square neighborhood as Ωid with center of pixel i and radius of t, and let the intensity value of any pixel in the neighborhood be d_i,j, j∈Ωid(8)r¯(i)=1m∑j=1mri,j(9)d¯(i)=1m∑j=1mri,j

Where m = (2t + 1)².

Based on Eq. (6), the value of the upper threshold at each pixel in an image can be adaptively determined according to the image content. Then, the adaptively truncating gradient is obtained by formulas (4) and (5). Figure 1 shows the gradient map and the adaptively truncating gradient map corresponding to the reference image and the distorted image. It can be seen that the maximum amplitude of the gradient map is approximately 250, while the maximum amplitude of the adaptively truncating gradient is approximately 70.

Figure 1.

With the adaptively truncating gradient defined, the local quality of the distorted image is predicted by the similarity between the adaptively truncating gradient of r and d, which is defined as(10)S(i)=2GT(r,i)⋅GT(d,i)+CGT2(r,i)+GT2(d,i)+C

Where the parameter C is introduced to avoid the denominator becoming zero and supplies numerical stability. The range of S(i) is from 0 to 1. Obviously, on the one hand, S(i) is close to 0 when G_T (r, i) and G_T (d, i) are quite different. On the other hand, S(i) will achieve the maximal value 1 when G_T (r, i) is equal to G_T (d, i).

The overall quality score of the distorted image is predicted by the local quality S(i), which is calculated as follows :(11)score=1N∑i=1NS(i)

A higher score indicates better image quality.

All the experiments in this study were implemented in MATLAB R2016b and executed on a Lenovo Ideapad700 laptop with Intel Core i5-6300HQ@2.3-GHz CPU and 4 GB RAM. Several well-known FR metrics were used when comparing performances with the proposed method, including PSNR, SSIM[1], FSIM [2], GMSD[3], DASM[5], IFC [8], VIF [9], MS-SSIM [10], and SSRM [11]. To widely evaluate the performance of these metrics, six public databases were employed for the experiments: TID2013 [12], TID2008 [13], CSIQ [14], LIVE [15], IVC [16] and A57 [17]. The TID2008 database consists of 25 reference images and a total of 1700 distorted images, each of which is distorted using 17 different types of distortions at four different levels of distortion. The TID2013 is an expanded version of TID2008, which contains 3000 distorted images with 24 distortion types. The LIVE database includes 29 reference images and 779 distorted images with five distortion types. The CSIQ database contains 30 original images and 886 distorted images degraded by six types of distortion. The IVC database consists of 10 reference images and 185 distorted images. The A57 database includes 3 reference images and 54 distorted images. Note that for the color images in these databases, only the luminance component is evaluated.

Four commonly used performance criteria are employed to evaluate the competing IQA metrics. The Spearman rank order correlation coefficient (SROCC) and Kendall rank order correlation coefficient (KROCC) are adopted for measuring the prediction monotonicity of an objective IQA metric. For compute the other two criteria, the Pearson linear correlation coefficient (PLCC) and the root mean squared error (RMSE), we need to apply a regression analysis. The PLCC measures the consistency between the objective scores after nonlinear regression and the subjective mean opinion scores (MOS). The RMSE measures the relative distance between the objective scores after nonlinear regression and MOS. For the nonlinear regression, we used the following mapping function:(12)QP=β1[12−11+eβ2(Q−β3)]+β4Q+β5

where Q and Q_P are original objective scores of an IQA metric and the objective scores after regression, respectively. β_i, i = 1, 2, ⋯, 5 are the fixed parameters. Higher values of SROCC, KROCC, PLCC and lower RMSE values indicate a better performance of IQA metrics.

For the proposed metric, there are three parameters that need to be set to obtain the final quality score. They are T₀, t and C. Selecting the first 8 reference images and corresponding 544 distorted images in the TID2008 database as the testing subset, we choose the parameters that can yield the highest SROCC. The result is T₀ = 3, t = 51 and C = 1600.

To further analyze the effect of threshold parameter T₀, more experiments were carried out. Figure 2 shows the SROCC performance with different T₀ values on six databases. On most databases, SROCC can is best when T₀ is 3. This result indicates that the range of upper threshold T is approximately [0,255/3] for an 8-bit grayscale image according to formula (6). If the change in image intensity is above 255/3, then it will be masked in visual perception.

Figure 2.

Table I lists the SROCC, KROCC, PLCC and RMSE results of ten metrics on six databases, and the two best results of each row are highlighted in bold. Overall, the methods which employed the gradient feature performs well across all the databases, such as FSIM, GMSD, DASM and the proposed metric. This partly demonstrates the validity of considering the degradation of gray changes in quality evaluation. Furthermore, the proposed metric performs well, outperforming SSIM and SSRM and competing with FSIM and GMSD.

Among the six databases, TID2013 has the highest number of distorted types. Table II lists the SROCC results of ten metrics about each individual distorted type of the TID2013 database. The proposed algorithm performs well in variety of distortion types. In particular, the proposed algorithm is outstanding for JPEG, JP2K and JPEG-trans-error distortion types that are sensitive to variations.