Research on Multimodal Image Tampering Detection and Counterfeit Image Recognition Techniques under Deep Learning Framework

The steps of the feature recognition process are as follows.

In the first step, the RGB color image is inputted, and the YCbCr transformation is performed on it, from which the Cr channel component is extracted and used for the next step. In the second step, for the generated Cr channel component, the BDCT transform of 8×8 image blocks is performed, i.e., the Cr channel component is sliced into a number of 8×8 pixel image blocks, each of which is adjacent to and not intersecting with each other. In the third step, a 2-dimensional DCT transform is performed on all 8 × 8 pixel image blocks to generate 8 × 8 pixel DCT image blocks. In the fourth step, for each 8×8 pixel DCT image block, LBP transform is performed to generate an equal number of 8×8 pixel LBP image blocks. In the fifth step, the mean value of the pixels at the corresponding positions of all the LBP image blocks is calculated, and the 8 × 8 pixels are able to generate a 64-dimensional feature vector, which represents the initial input RGB colour image. The basic flow is shown in Fig. 1.

Figure 1.

Faking image data processing flow chart

The original colour flow takes the original tampered image as the model input, and in this paper, we improve the feature extraction method in the original Faster R-CNN and learn the features in the original tampered image by using the deep residual network, ResNet50, and the Recursive-FPN, a recurrent feature pyramid network, which the Google Research Institute proposes. Among them, Recursive-FPN is an improvement of the original FPN network, which recirculates the features output from the ordinary FPN and inputs them to the backbone network, increasing the global features and sensory field to effectively acquire multi-scale information, thus improving the learning effect of the model.

In order to make the model pay more attention to tampering with some of the features and alleviate the problem of weakening the ability to distinguish global features between classes due to the lack of tampering information, this paper improves the original RPN network and designs an RPN network combining spatial attention mechanism and channel attention mechanism (CBAM-RPN). Firstly, the attention feature map is taken as the input of the RPN network, and then CBAM infers the attention weights from the spatial dimension and the channel dimension sequentially and then multiplies the attention weights with the corresponding elements of the feature map to adaptively adjust the features. The process is shown in Fig. 2.

Figure 2.

Structure of the proposed model

In this paper, the CBAM module is added in front of the RPN network in the primary colour branch and the process can be expressed as equation (13): 13Pi=Ci⊗Mi(Ci)⊕fupsmpping2×2(Pi+1) Where C_i denotes the output features of the ResNet extraction phase i, the M_i denotes the attention module, operation symbol ⊕ denotes the addition of the corresponding elements in the feature map, operation symbol ⊗ denotes the multiplication of the corresponding elements in the feature map, fupsamping2×2 denotes an upsampling operation with convolution size 2 × 2, and p_i denotes the final generated feature map.

The RPN network in the original colour flow generates regions of interest using features that have been attentively extracted for the purpose of bounding box regression.The loss function L_R of the CBAM-RPN network is defined as shown in Equation (14): 14LR(gi,fi)=1Nc∑iLc(gi,gi*)+λ1Nr∑igi*Lr(fi,fi*) Where i denotes the i nd anchor frame, g_i denotes the predicted probability that there is a potential tampering region in the i th anchor frame, g_i^* denotes the true label of the ith anchor frame, f_i and fi* denote the boundary coordinates and label of the i th anchor frame, respectively, L_c denotes the loss of the crossover of the CBAM-RPN network, L_r denotes the regression loss used to smooth the predicted bounding box, denoted by Smooth L₁, and N_c denotes the number of small batches in the CBAM-RPN network with small batch size, N_r denotes the total number of anchors, λ is the balance coefficient, which is set λ = 10 in this paper. Where the Smooth L₁ loss function is shown in Equation (15): 15SmoothL1(x)={ 0.5x2,| x |<1| x |−0.5,otherwise

The manipulated image may undergo some post-processing that may hide copying or splicing boundaries and reduce contrast differences. At this point, enhancement of boundary artifacts using gradient filtering can provide additional evidence of tampering. Gradient streaming is designed to perform edge detection on the input image and feature learning on anomalous edges. The principle of gradient streaming to provide additional tampering traces is as follows: if consecutive pixels of an image have discontinuous gray values, it indicates a local discontinuity, and this discontinuity becomes an edge element. If adjacent edge elements can be joined into a line segment along their tangent direction, the segment is called a boundary. Boundaries reflect the physical extent occupied by an object or region in an image and are a useful and important feature.

This paper aims to detect the edges at the periphery of an object and eliminate the impact of external noise on the experiment. Based on the experimental results and related literature, and comparing the advantages and disadvantages of different edge detection operators, the Gaussian-Laplace operator LoG is chosen in this paper for edge detection in gradient flow.

LoG filter is a combination of the Gaussian filter and the Laplace filter. In order to prevent the Laplace filter from calculating the quadratic differential to make the noise of the image more obvious, the filtering process first uses a Gaussian filter to smooth the image and then uses the Laplace filter to outline the image clearer [28].

Analysed from the theoretical point of view, the biggest advantage of LoG operator is that it is an isotropic edge detection operator, which has the same enhancement effect on edges in any direction, is insensitive to orientation, and has the ability to resist noise, which will not appear multiple non-edge pixel responses in the process of edge detection, and avoids pseudo edges to reduce the recognition accuracy of the tampering traces in the image. In the gradient flow, the LoG operator is used in the following process: firstly, the input image I is subjected to smoothing operation by Gaussian distribution factor σ, and then the edges are detected using the Laplace operator as shown in Eq. (16) and Eq. (17): 16Gσ(x,y)=12πσ2exp(−x2+y22σ2) 17Δ| Gσ(x,y)*I(x,y) |=| ΔGσ(x,y) |*I(x,y) Where G_σ (x, y) denotes the 2D Gaussian function, (x, y)is the image point coordinates, and f (x, y) denotes the pixel value corresponding to point (x, y). The LoG operator is equivalent to the derivation of the Gaussian function, and then the convolution operation of the original image, and the final result can be expressed as equation (18): 18LoG=ΔGσ(x,y)=∂2Gσ(x,y)∂x2+∂2Gσ(x,y)∂y2

Assuming that I(i, j) is the pixel value corresponding to edge point (i, j), Eqs. (19) to (22) describe the deviation between the pixel value of this edge point and the pixel value of the neighbouring point.

Horizontal direction: 19h(i,j)=| lg |I(i,j)−I(i−1,j)I(i,j)−I(i+1,j)‖

Vertical: 20v(i,j)=| lg |I(i,j)−I(i,j−1)I(i,j)−I(i,j+1)‖

Diagonal direction: 21d1(i,j)=| lg |I(i,j)−I(i−1,j−1)I(i,j)−I(i+1,j+1)‖ 22d2(i,j)=| lg |I(i,j)−I(i−1,j+1)I(i,j)−I(i+1,j−1)‖

For a normal edge point, the deviation between the two sides is not significant and the difference between the two sides of the edge point is relatively small, so the value of the computation factor (h, v, etc.) is correspondingly small. However, for a tampered edge point, the two sides tend to be anomalous, and the edge point is too biased to one side resulting in a correspondingly larger calculated result. Factors for the 4 directions of the edge point are calculated, and if the value of one factor is greater than a threshold value of T, it is considered a suspicious edge point. Where, the threshold value T is an empirical value obtained through experiments.

Bilinear pooling can fuse two-stream deep convolutional networks to improve the confidence of detection while preserving the spatial information of CNN networks. In order to save memory and speed up training without degrading performance, in this paper, we use compact bilinear pooling to fuse the features of the original colour stream and the gradient stream. Denote the output of compact bilinear pooling by x, which is defined as shown in equation (23): 23x=fcTfg Where f_c denotes the original colour flow network feature and f_g denotes the gradient flow network feature. The fused features x are used as the basis for the subsequent judgement of whether the region has been tampered with.

The overall loss function L_t of the model in this paper is defined as shown in equation (24): 24Lt=LR+Lc(fc,fg)+Lb(fc) Where, L_R denotes the CBAM-RPN network loss, L_c denotes the cross-entropy classification loss which is used to evaluate the classification of the last tampered region, which is jointly determined by the original colour flow network feature f_c and the gradient flow network feature f_g, and L_b denotes the bounding-box regression loss, which is used to evaluate the regression of the bounding-box of the last tampered region using Smooth L₁, which is determined only by the original colour flow network feature f_c.

The complete flow of the model proposed in this paper is shown in Fig. 3.

Figure 3.

Workflow of the proposed model

In order to compare the feature recognition effect of the forged image feature recognition methods in the face of training sets with different data volumes, the experiments in this paper are carried out using three datasets containing different data volumes such as NIST12, CASIA and IMD2020, the above datasets contain 581, 974 and 2022 tampered images respectively. Algorithms such as MFCN, BusterNet, and RGB-N Net are selected as comparison algorithms for the experiments.

The experimental results of each model on different data volume data are shown in Table 1. With the increase of data volume, the classification accuracy of each model increases while the time of each round of the model training process grows. In this paper, the classification accuracy is the highest in all the experiments on different data volumes, with 74.81%, 80.07%, and 83.92% classification accuracy in the three training sets, respectively.

Figure 4 shows the comparison of model training efficiency. Fig. 4(a)-(c) shows the recognition accuracies of the different models on the NIST16, CASIA and IMD2020 datasets, thus validating the experimental results in the previous section. Fig. 4(d) shows the comparison of the training efficiency of each model with different data volume experiments.

Figure 4.

Comparison of model training efficiency

When the data volume is 581 frames of small data, the training speed of each model is faster, and the training time of each round is within 28 seconds, which is not a big difference. When the data volume is 974 frames of medium-sized data, the training speed of each model starts to show the difference. The RGB-N Net method has a longer sequence length of the input vector sequences, which can result in a disadvantage due to the larger computational volume of its convolution process. The training efficiency of YCrCb-N Net starts to slow down as the data volume increases due to its large number of parameters. The difference in training efficiency between this model and MFCN at this time is not significant, but this model achieves better classification results with less training time.

In order to verify the effectiveness of the image tampering detection model proposed in this paper, experiments are conducted on the datasets CASIA19 and NISIT21 and compared with some other baseline methods.

The dual-stream Faster R-CNN model with improved features can both classify tampering methods and locate tampered regions. Most of the existing algorithms can only judge whether the image is tampered or the location of tampered regions. In this paper, the RGB-Net algorithm is selected, which can simultaneously identify the tampering method and locate the removal of the tampering. The MFCN method is selected, which uses a convolutional neural network to tamper with the localisation of image stitching detection. BusterNet is selected, which is used for copy-move detection and localisation. Also, the performance of YCrCb-N Net and RGB-ImprovementN Net for tampering image detection is tested in this paper.

Table 2 shows the comparison of F1 scores of various algorithms on the two datasets. It can be seen that the improved dual-stream Faster R-CNN model proposed in this paper can significantly improve the performance of localising tampered regions of an image compared to the dual-stream Faster R-CNN algorithm based on RGB stream and noise stream. In addition, the dual-stream Faster R-CNN algorithm can achieve about the same performance as the single tamper detection localisation (splicing, copy-movement). However, the improved dual-stream Faster R-CNN algorithm has better tamper region localisation performance than MFCN and BusterNet, which illustrates that the use of the original colour stream and the improved gradient stream can improve the localisation accuracy of the algorithm.

Meanwhile, using attention-extracted features to generate regions of interest and selecting LoG for edge detection in gradient flow can improve the algorithm’s localization effect on tampered regions. And the improved gradient flow improves the model performance more obviously, especially on the NISIT21 dataset, which indicates to a certain extent that the improved gradient flow can improve the model performance in detecting tampered images that undergo rotational transformations.

In addition, it can be found that the detection performance of the four algorithms is better than that of the NISIT21 dataset on the CASIA19 dataset, which may be due to the fact that the sample size of the CASIA19 dataset is much larger, and the tampered part of this dataset has undergone pos-tprocessing such as geometrical transformations to conceal the tampering traces, which affects the algorithms’ detection performance.

Table 3 shows the comparison of the average recognition accuracies (average of the results on two datasets) of the algorithms on two datasets for the three different tampering modalities of image splicing, copying, moving, and removal.

It can be concluded that the improved dual-stream Faster R-CNN algorithm achieves the best results for all three tampering methods in recognition. The dual-stream Faster R-CNN algorithm is slightly inferior to the MFCN method (0.968) in recognition of splicing. However, the method is able to achieve recognition of multiple types of tampering modalities and achieves about the same performance as single image tampering. In addition, the improved dual-stream Faster R-CNN algorithm proposed in this paper, compared with the initial dual-stream Faster R-CNN algorithm, achieves an improvement of 2.7 percentage points for splice detection, 2.7 percentage points for copy-shift detection, and 3 percentage points for removal tampering detection in terms of the average accuracy of detection.

In real-world scenarios, tampered images are often subjected to a wide variety as well as different levels of post-processing. In order to assess the robustness of the proposed model, this paper uses a benchmark training mode and evaluates the performance of the model under a variety of post-processing attacks of different intensities on the IMD2020 and NIST16 datasets.

These post-processing attacks include scaling (adjustment rates of 0.35× and 0.75×), Gaussian blurring (kernel parameters of 4 and 16), Gaussian noise (standard deviation of 4 and 16), and JPEG compression (quality factors of 60, 80, 90, and 120). To ensure fairness, all models were trained and tested according to the same experimental settings. The models in this paper were compared with MVSSNet and IF-OSN. The experimental results on the IMD2020 dataset are shown in Table 4.

Where the bold font indicates the highest value, the data show that on the IMD2020 dataset, this paper’s model significantly outperforms the other 2 models in the face of a variety of post-processing attacks with different strengths, and the F1, IoU, and AUC metrics show better results, especially under the attack of Gaussian noise (16), the F1 metrics of this paper’s model drop by 58%, while those of MVSSNet and IFOSN drop by 97% and 85%. This indicates that the model in this paper is more resistant to Gaussian noise.

The experimental results on the NIST16 dataset are shown in Table 5. The F1, AUC, and IoU metrics of this paper’s model are significantly better than those of the MVSSNet model on the NIST16 dataset. Compared with IF-OSN, the initial F1 and AUC metrics of this paper’s model are lower, but the model performs better in the face of Gaussian noise, Gaussian blur, JPEG compression (60), JPEG compression (80), and scaling (0.35×) attacks. Under JPEG compression (90), JPEG compression (120), and scaling (0.75×) attacks, the performance of this paper’s model and the IF-OSN remains basically the same. The analyses above illustrate the reliability of this paper’s model in dealing with localization tampering.