Research on the Reconstruction Technology of Digitized Artworks Based on Image Processing Algorithms and Its Cultural Inheritance Value

The main idea of generative adversarial networks is to think of generative and discriminative networks as two opposites [27]. Their two respective functions are similar to the counterfeiter and the police, while the generative network is the role of the counterfeiter, the discriminative network is the role of the police, the counterfeiter continues to create fake images, the police will continue to screen out fake images, and they continue to cycle between the counterfeiter in order to allow the police to screen out fake images, he will have to improve their own counterfeiting techniques, and by the same token, the police to continue to screen out fake images, also have to improve their screening ability, when the police screen out fake images, and when the police screen out fake images, also have to improve their screening ability, and the police have to improve their screening ability. Similarly, if the police want to continue to screen out fake images, they also have to improve their screening ability, when the police can not screen out which images are fake, it proves that at this time the counterfeiting technology of the counterfeiting personnel is the best, that is to say, at this time, the generative network generates the highest quality images. The corresponding objective function is calculated as shown in equation (1): (1)minGmaxDV(G,D)=Ex~Pdaaa(x)logD(x)+Ez~Pmax(z)log(1−D(G(z)))

Where, P_data is the probability distribution of the real data, x is the real image image, P_noise is the probability distribution of the generated data, z is the generated image and E is the expectation.

The Pix2Pix network is based on conditional generative adversarial networks, and conditional generative adversarial networks are still distinguished from generative adversarial networks in that it not only learns the mapping of the initial values z to the real image, but also observes the mapping of the original image x to the real image [28]. It should be noted that Pix2Pix networks are trained using paired datasets. The objective function of the conditional generative adversarial network is shown in Equation (2): (2)G*=argminGmaxDLcGAN(G,D)+λLL1(G)

where G is the generative network, D is the discriminative network, min_G is G to minimize the objective function, and max_D is D to maximize the objective function.

The network structure of Pix2Pix still uses the combined form of generative and discriminative networks, both of which are in the form of convolution plus batch normalization and ReLU activation. In order to solve the problem of generative networks having bottleneck layers, the jump connections in U-Net networks are finally used, mainly in the form of, adding jumpers between each i-layer and n-i-layers, where each jumper can simply connect the feature channels of the i-layer and n-i-layers together, where n is the total number of layers in the network.

Like the Pix2Pix network in the previous section, this kind of network can achieve a good conversion effect, but there is a very obvious defect, that is, it can only be used in pairs of datasets, if the use of ordinary datasets, the conversion effect is not ideal, this situation is equivalent to the supervised learning, but this kind of pairs of datasets to obtain not only the idle difficult, but also more expensive.

If you modify the sentence structure of a sentence and convert a “put” word into a “be” word, and then the “be” word into a “put” word, then the resulting “put” word should be the same as the original “put” word. In addition to the adversarial loss, the cyclic consistency loss is also added, and the objective function is calculated as shown in equation (3): (3)G*,F*=argminG,FmaxDX,DYL(G,F,DX,DY)

In this, generative network G converts the image from X domain to domain, generative network converts the image from Y domain to X domain, discriminator D_X is used to determine whether the input image is an image in X domain and discriminator D_Y is used to determine whether the input image is an image in Y domain.

CycleGAN uses a generative network architecture that starts with two convolutional layers with a step size of 2, and two transposed convolutional layers with a step size of 1/2, followed by 9 residual blocks connected to process images of size 256*256, and uses instance regularization to improve the quality of the style migration of the feedforward network [29]. The discriminative network uses a 70*70 PatchGAN structure where each block comes to determine whether the image is real or not, this structure has fewer parameters and can handle images of any size.

StarGAN is a new generative adversarial network that requires only a generative network and a discriminative network for multiple domain transformations. For images, attributes are defined for the features inherent in the image, for example, the attributes of a person’s eyes can have single eyelid and double eyelid, etc., and the attributes of the nose can have a high and collapsed nose, and the domains actually represent a set of images that have the same attributes, so that black hair represents one domain and yellow hair another. StarGAN takes the information of the images and domains together as inputs, and it mainly learns How the input images can be flexibly transformed to the corresponding domains and use the labels to represent the information of the domains.

The calculation of loss function of generative network is shown in equation (4) and the calculation of loss function of discriminative network is shown in equation (5): (4)LD=−Ladν+λclsLclsr (5)LG=Ladν+λclsLclsr+λrecLrec

Where, L_D is the loss of the generative network, L_G is the loss of the generative network, L_adv is the adversarial loss, L_cls is the categorization loss, λ_cls is the categorization loss weight, L_rec is the recall loss and λ_rec is the recall loss weight.

StarGAN’s image transformation performs better for face attributes and expressions.

Considering that an image domain contains several images, we use the average image entropy of all images in each image domain as the image entropy value for that domain by calculating the average image entropy of all images in that domain [30]. The image entropy of an image domain is defined as follows: (6)mEntropy=1N∑n=1NEntropy(imgn)

Where N denotes the total number of images in an image domain, img_n denotes the image instances in the image domain, and Entropy(*) denotes the image entropy value calculation on the input image, the detailed definition formula is shown in Equation (7): (7)Entropy=−1c∑c255∑j=0255P(c,i,j)×logP(c,i,j)

where c denotes the channel index of the HSI color space channel, i and j denote the length and width positions of the image, and P_(c,i,j) denotes the probability of the number of occurrences of the pixel at the corresponding position over the total number of pixels.

In order to measure the image entropy difference between the two image domains in the image conversion task more intuitively, we calculate the information entropy ratio between the two image domains, which is defined in detail as follows: (8)EntropyRatio=max(mEntropy(X),mEntropy(Y))min(mEntropy(X),mEntropy(Y))

where X and Y denote different image domains in an image conversion task.

In order to better measure the information asymmetry of the artwork style migration task, we calculated the information entropy ratios of several generic image conversion tasks at the same time. It can be found that the information entropy ratios of both the artwork style migration task and the asymmetric image conversion task are larger than the information entropy ratio of the symmetric image conversion task, which indicates that the artwork style migration task is characterized by the asymmetry of the domain information richness, and it can be regarded as an asymmetric image conversion task.

The asymmetric loop consistency structure network consists of two generators, two discriminators and a saliency edge extractor (SEE). Generator G is a residual block-based structure for style migration from real nature photos to Chinese art works image domain; Generator F is a dense block-based structure for style migration from Chinese art works to real nature photos image domain. Two discriminators D_X and D_Y are used to determine the authenticity of the generated real nature photos and the generated art work images, respectively. The saliency edge extraction module was used to extract the saliency edge maps in the images. 1)

Generators

Based on the symmetric cyclic consistency structure of Cycle GAN, we targeted to improve the structure of generators G and F for the information richness asymmetry characterizing the Chinese artwork style migration task. We designed Generator F with Dense Block as the core transformation module, which has stronger feature characterization capability than Generator G with Residual Block as the core transformation module, and can effectively match the transformation difficulty from the artwork image domain with low information complexity to the real natural image domain with high information complexity.

The generator in CycleGAN consists of an encoder and a decoder, and the middle contains a residual block-based conversion module that determines the characterization ability of the whole generator. Considering that the dense connection mechanism used by dense blocks has the advantage of enhancing feature propagation and encouraging feature reuse, replacing the residual blocks in the conversion module with dense blocks can effectively improve the characterization ability of the generator.

Based on the original loss of CycleGAN, we introduce feature level based cyclic consistency loss and saliency edge loss. 1)

Adversarial loss

The main role of the adversarial loss is used to optimize the generator and the discriminator so as to improve the generative ability of the generator and the discriminative ability of the discriminator. For the style migration from the real natural image domain to the direction of the artwork image domain, the use of the adversarial loss constraint model can generate an output image that is closer to the artwork image. The detailed calculation of the adversarial loss is shown in equation (9): (9)lossGAN(G,DY,X,Y)=Ey~YlogDY(y)+Ex~Xlog(1−DY(G(x)))

where X and Y represent the data distributions in the real natural photo domain and Dunhuang mural image domain, respectively. An identical confrontation loss loss_GAN(F, D_X, X, Y) is applied to the style migration from the artwork image domain to the direction of the real natural image domain, which is used to constrain the model-generated images to be closer to the real natural image domain. 2)

Identity Loss

The identity loss function can help the model to avoid meaningless transformations, and at the same time constrain the model to keep the same color distribution of the input image and the output image, the detailed calculation is shown in Equation (10): (10)lossIdentity(G,F,DX,DY)=Ex~X[||F(x)−x||1]+Ey~Y[||G(y)−y||1]

Where VGG_{com4_4}(*) represents the output feature of the conv4_4 layer in the pre-trained VGG network, which can effectively characterize the content structure information of the image. 4)

Significance edge loss

We propose a saliency edge loss to simulate the stylistic feature of prominent body strokes in artworks. Firstly, the real natural image and the generated artwork image are input to the saliency edge module, and then the obtained saliency subject edge map is balanced cross-entropy loss calculation, so as to obtain the saliency edge loss, and the detailed formula is shown in Equation (12): (12)lossSalient−Edge(G,X) =Ex~X[−1N∑n=1NαE(x)nlog(E(G(x))n)+(1−α)(1−E(x)n)log(E(G(x))n)]

Where N is the total number of pixels in the input image corresponding to the saliency subject edge map, E(·) is the edge extraction model HED network, and α is a balanced weight factor. 5)

Total target loss

Finally, the total objective loss of the model is calculated in equation (13): (13)L(G,F,Dx,DY,X,Y)=λ1lossGAN(G,DY,X,Y)+λ2lossGAN(G,DX,X,Y) +λ3lossCycle(G,F,X,Y)+λ4lossIdentity(G,F,X,Y)+λ5lossSalientEdge(G,X)

We aim to optimize the following objective function as detailed in Eq. (14): (14)G*=argminG,F,DX,DYL(G,F,DX,DY,X,Y)

The art style migration model proposed in this paper introduces cyclic consistency loss and saliency edge loss. In order to analyze the improved CycleGAN loss in this paper, the μ value is set to 0, 1, 5, 10, 20, and 50, respectively, where the change of the μ value represents the proportion of the cyclic consistency loss and the saliency edge loss in the overall loss function. As the μ value increases, the change in cyclic consistency loss is specifically shown in Figure 1. The overall trend of the change in the cyclic consistency loss is first a rapid decrease, and then the whole trend slows down. Such change characteristics indicate that the source domain to target domain converted image again converted to the source domain has better results with the increase of the μ value, i.e., the generated image restored back to the color and feature changes are smaller, and its texture information is lost relatively less.

Figure 1.

Change of the consistent loss of the cycle

As the value of μ increases, the change trend of the significance edge loss is specifically shown in Figure 2. It can be seen that the whole trend of the saliency edge loss is accompanied by a larger value of μ to become smaller, and its overall trend is a rapid decrease to leveling off. This helps the generated image to learn more features of the transformed target domain while the irrelevant domain features of the source domain can be preserved as much as possible.

Figure 2.

Change of significant edge loss

When the loss is increased, the classification accuracy of the images is shown in Table 1. From the experimental results in the table, it can be seen that increasing the loss, by adjusting the loss to the appropriate scale hyperparameter μ, can improve the realism of the generated images, and the image classification accuracy of the monet2photo dataset, the horse2zebra dataset, and the summer2winter dataset can reach a maximum of 0.91 (μ =20), 0.87 (μ =5), and 0.87 (μ =10). However, the test results also show that the image classification accuracy of monet2photo dataset, horse2zebra dataset, and summer2winter dataset can be 0.78(μ =50), 0.72(μ =20), and 0.73(μ =20) at the lowest. Obviously, if μ is not selected appropriately, it will cause some images to be converted to a poor degree and the image classification accuracy will decrease. Therefore, the appropriate loss ratio hyperparameters should be selected according to the dataset and practical application requirements.

1)

Subjects

Combined with the existing eye movement experiment related research, college students have a certain aesthetic judgment ability, and the experimental data obtained by choosing them for the eye movement experiment is representative and credible. In this experiment, 50 college students were randomly selected as test subjects.

In order to ensure the validity of the experiment, the participants were required to have normal visual acuity or corrected visual acuity, and were not allowed to wear colored contact lenses during the experiment, or the results would be affected.

The subjects were invited to view the three digitized artworks reconstructed by the art style migration model of this paper and the traditional Pix2pix method, respectively, and the eye movement data obtained were collected as shown in Table 4. It can be seen that the subjects’ average gaze time reached 14.89s and the average number of gaze times was about 64 when viewing the digitized artworks reconstructed using the method of this paper. The average gaze time and the average number of gaze times obtained by the digitized artworks reconstructed using the traditional Pix2pix method, on the other hand, were 9.78s and 20 times, respectively. Among different types of digitized artworks, the average gaze duration of subjects viewing the digital painting artworks, digital sculpture artworks, and digital craft artworks reconstructed in this paper were 12.96s, 16.44s, and 15.27s, respectively, and the average number of times they looked at the artworks were about 60, 74, and 50 times, respectively, which were higher than that of the traditional Pix2pix method. In terms of both the average duration and the number of gaze times, the eye movement data of this paper’s method still significantly outperforms the traditional Pix2pix method in the reconstruction of different types of digitized artworks.

The reconstruction of digital artworks not only ensures that the connotation of art remains unchanged, but also opens up the public’s aesthetic vision and increases the depth of cultural connotation dissemination. 3R digital art can rely on the exhibition hall to build an embodied immersive scene based on digital artworks, allowing the audience to change from passive “static viewing” to active “wandering”, from relying on their own aesthetic awareness to independent activities in the “real” three-dimensional exhibition space, emotionally accessing the appreciation process, and cooperating with multiple perceptions of vision, hearing, touch, taste, and smell, leading the audience to walk in the virtual space and feel the beauty of the artwork.

Works of art have a strong aesthetic education and the function of cultivating people’s cultural temperament. Digital art works are art forms catering to the cognitive habits of modern people, so that the viewer does not need to deliberately withdraw from the busy life, you can always taste from the fun of interaction in the elegant mood, calm charm and other artistic infectious atmosphere, in the paintings to find a pure land, a hint of relaxation. The combination of the two can pass on and promote art and culture to a greater extent, and the cultivation of culture and artistic aesthetics can help the public find themselves in the world, adjust their mentality, and invisibly change the public’s perspective on society and the natural environment.

Art theory and appreciation courses are inevitably raw and not easy to understand if they only rely on teachers’ verbal explanations. The combination of virtual and real scenes brought about by the reconstruction of digital artworks can make the educational scene more vivid and lively, break through the limitations of textbooks, PPTs and other flat teaching methods, and hand the classroom over to the students for independent exploration and communication. While activating the inner vitality of the artwork, it can also stimulate students’ interest in traditional culture, let students enter the art world, give full play to the idea of “learning by understanding”, feel the spirit and creative intentions of the artist, reach a spiritual communication across space and time, and get to know the works of art at a deeper level and understand the stories behind them, so as to realize the knowledge and emotion in teaching better. In order to better realize the goal of knowledge and emotion in teaching, and to guide students to form a correct aesthetic view.

The value of art works is also reflected in the practical level, the unique traditional aesthetics enriches the thinking of art design workers, and the characteristics of digital art works can make the connotation of the art works itself extend in the direction of painting, games and so on. For example, in the field of painting, digital artwork reconstruction technology can be used to simulate the painting brush techniques, hue changes, to create vivid and interesting painting effects, to form a digital artwork based on traditional painting heritage, in line with the spirit of the times and culture, to bring a fuzzy medium of the painting experience, to create a more immersive space for the creator of the painting.