Research on traditional oil painting innovation and creation technology assisted by artificial intelligence algorithm

Oil painting is the creation of an artist or an individual through his or her imagination and conception of objective things or abstract thoughts. When external things enter the artist’s creative inspiration area, the artist first recognizes and perceives different scenes, and then carries out different degrees of segmentation of the relevant scenes, and then determines the specific position of each object, and in which area of the painting each object should be. The process of generating the oil painting is shown in Figure 1.

Figure 1.

The process of oil painting

1)

Color Statistics

Usually, the color distribution in natural images is not balanced, while the color distribution in oil paintings is more harmonious. In this paper, blue in the traditional sense is specified as the cold pole, while orange in the traditional sense is specified as the warm pole. In RYB space, the psychological color temperature is expressed as: (1)Colortemperature=Saturation*sin[Hue]/{[Saturation*cos(Hue)^2+1]} $$Color\,temperature = Saturation*sin[Hue]/\left\{ {[Saturation*cos(Hue\hat )2 + 1]} \right\}$$

In this case, as the saturation of the color changes, so does the color temperature. In terms of psychological color temperature, oil paintings show a tendency to be on the warm side. In terms of saturation, the distribution of oil paintings is relatively even. 2)

Placement of brushes

The specific operation process in the process of brush placement: first of all, it is necessary to select a bunch of brushstroke dictionaries, which are required to have different colors, different sizes and various shapes from each other. Then, these brushstroke dictionaries are placed on the canvas for oil painting, their total energy is calculated, and they are placed in the order of the total energy from the largest to the smallest, thus forming a link. Repeat the process, arranging the brushes in order, from largest to smallest, until the brush with the smallest stroke value is placed. In the process of placing brushes, summarize to get the model about the placement of brushes.

Figure 2.

Mean Shift diagram

Also, it is possible to redefine thermal diffusion in diffusion technology, which can be defined as a drop in energy or a cooling process of energy. There are two main forms of equations concerning energy, one of which is the continuous form of the equation (Eq. 5) and the other is the discrete form of the equation (Eq. 6): (5)ε(u)=∫x,y|∇u|2dxdy (6)ε(u)=12∑k(uk+1−uk)2

According to the strategy related to gradient descent, it is easy to find out the process about the energy descent, which leads to the discrete solution of the equation. One point to note is that thermal diffusion and directional diffusion are quite different and cannot be applied to each other.

The environment of the painting simulation mainly consists of a target image I, a drawing board C, and a neural renderer R, with the initial state of the board being C₀. Given a finite number of steps n, at each step, the renderer renders a new state of the board, C_t+1, i.e., C_t+1 = R(C_t, a_t), based on the current state of the board, C_t, and the received stroke control parameters, a_t, where t is the number of the current steps, and the goal of the intelligent is to find a set of sequences of stroke control parameters, a₁, a₂, ..., a_n, such that the the final rendered drawing board C_n is as close as possible to the target image visually. This process is modeled as a Markov decision process with state space S, action space A, state transfer function trans(s_t, a_t), and reward function r(s_t, a_t). 1)

State and state transfer function

In reinforcement learning state represents the information observed by the intelligence from the environment. Here the state is divided into three parts: the state of the drawing board, the target image, and the current step ratio, i.e., s_t = (C_t, I, t/n), where the state of the drawing board C_t and the target image I are represented by a 24-bit RGB image with the image size of H × W × 3, and the step ratio t/n is a decimal number that provides information about the number of remaining steps, which facilitates the planning by the intelligent body. The state transfer function s_t+1 = trans(s_t, a_t) defines how the drawing board is transferred between states from this step to the next, which is partially known due to the use of the neural renderer R.

where r_t denotes the reward obtained by the intelligent in step t, d_t denotes the distance between C_t and the target image I, and d_t+1 denotes the distance between C_t+1 and the target image I. In this paper, we draw on the idea of generative adversarial networks to calculate the distance between two images by constructing a neural network-based discriminator D: (8)d(C,I)=−D(C,I)

The input to the discriminator is a combination (C, I) consisting of a drawing board image and a target image, and the output D(C, I) of the discriminator is higher when the difference between the drawing board and the target image is smaller, and the output D(C, I) of the discriminator is lower when the difference between the drawing board and the target image is larger.

1)

Model-based DDPG algorithm

When using Neural Renderer R, the state transfer function is partially known, i.e.,: (9)st+1=(Ct+1,I,t+1n)=trans(st,at)=(R(Ct,at),I,t+1n)

Meanwhile, the discriminator-based reward function is also known, i.e., in this paper, we improve the DDPG algorithm into a model-based DDPG algorithm by explicitly using these two known functions. First, the value network is used to approximate the state value function instead of the action value function, i.e: (10)V(st;WV)

where W^V is a parameter of the value network. The steps of the improved TD algorithm become: (1)

Use the strategy network to calculate the next action to be taken: (11)at+1=π(st;Wπ)

where α^V, W_now^V, W_new^V represents the learning rate, current parameters and new parameters of the value network, respectively.

Then, the method of following the parameters of the strategy network is improved, in the original DDPG algorithm, the strategy gradient is obtained by directly deriving the action value function, i.e.: (15)gt=∇WπQ(st,π(st;Wπ);WQ)

Here, the Bellman equation is applied before the derivation of the state value function, i.e: (16)gt=∇WπV(st;WV)=∇Wπ(r(st,at)+γV(st+1;WV))=∇Wπr(st,at)+γ∇WπV(st+1;WV)

As: (17)r(st,at)=d(Ct,I)−d(Ct+1,I)=d(Ct,I)−d(R(Ct,at),I)

Can be obtained by the chain rule: (18)∇Wπr(st,at)=∇Ct+1d(Ct+1,I)⋅∇WπR(Ct,at)

Ditto: (19)∇WπV(st+1;WV)=∇st+1V(st+1;WV)⋅∇WπR(Ct,at)

Hence the strategy gradient: (20)gt=∇WπR(Ct,at)⋅(∇Ct+1d(Ct+1,I)+γ∇st+1V(st+1;WV))

Continuing to use the chain rule to derive the strategy function gives: (21)gt=∇Wππ(st;Wnowπ)⋅∇aR(Ct,at)⋅(∇Ct+1d(Ct+1,I)+γ∇St+1V(st+1;WV))

The above equation has four main parts, which are: (22){∇Wππ(st;Wnowπ)∇aR(Ct,at)∇Ct+1d(Ct+1,I)∇St+1V(st+1;WV)

Since the policy function π, the renderer R, the distance function d and the value function V all use a neural network, the four derivatives above can be derived explicitly. Finally, the parameters of the policy network are updated using the following equation: (23)Wnewπ←Wnowπ+απ⋅gt 2)

Discriminators

In the framework of this paper, the discriminator used to compute the reward is also learnable. Let the parameters of the discriminator be W^D. For the data s_t = (C_t, I, t/n) drawn from the experience cache, this paper defines x = (C_t, I) as a fake sample and y = (I, I) as a real sample, thus defining the learning objective of the discriminator as: (24)maxWE[D(y;WD)]−E[D(x;WD)]

In this paper, a network structure similar to ResNet-18 is chosen as the feature extractor in the strategy network and value network. The inputs of the strategy network and value network are the states s_t = (C_t, I, t/n) (strategy network C = C_t, value network C = C_t+1) defined earlier, where C and I are both image data with the dimensions of H × W × 3. In this paper, the drawing board image, the target image, the step ratio, and the coordinate channel are spliced together to obtain an image tensor with the dimensions of H × W × 9 as the input data of the feature extractor.

In order to verify the ability of the DDPG neural renderer constructed in this chapter to generate the accuracy of oil painting strokes, the generative network of DCGAN (DCGAN-G), the UNet network, the PixShuffleNet structure, and the Dual-Pathway model (Dual-Pathway) are selected to constitute the neural renderers respectively in this subsection. Experiments will be conducted to compare the accuracy of the strokes generated by these five neural renderers as well as the perception of the image details of the generated oil paintings.

The A-E curves of the accuracy of randomly generated strokes by the DCGAN-G, UNet, PixShuffleNet and Dual-Pathway renderers and the DDPG renderer in this paper are plotted separately, all the renderers are trained at the same settings, where the horizontal axis is the training batch, and the vertical axis is the accuracy of randomly generated strokes. The results of the stroke generation accuracy comparison are shown in Figure 3.

Figure 3.

The accuracy of brushstroke generation

From the A-E curves plotted in Fig. 3, it can be seen that the weight parameters are continuously optimized at a training batch of 150, when the training accuracy of the neural network gradually increases. At around 250 rounds of training batch, the network gradually converges and stabilizes.

Among the five renderers, PixShuffleNet generates strokes with an accuracy of about 64.12%, the UNet renderer generates strokes with an accuracy of about 76.03%, the DCGAN-G renderer generates strokes with an accuracy of 80.29%, the Dual-Pathway renderer generates strokes with an accuracy of 77.04%, and the DDPG constructed in this paper network constitutes a renderer that randomly generates strokes with the highest accuracy of about 83.38%, which is better than other classes of renderers. The DDPG neural renderer constructed in this paper clearly has better generation capability.

In addition to the accuracy of the generated strokes, on the other hand, the operation efficiency of each model is evaluated using the computation time. The DCGAN-G, UNet, PixShuffleNet and Dual-Pathway renderers are used on RTX 3080Ti with the DDPG renderer of this paper for 2000 iterations on 10 experimental images, respectively. The data calculated by the two algorithms are shown in Table 1.

From Table 1, it can be seen that under the same configuration, the average value of the running time of DCGAN-G, UNet, PixShuffleNet and Dual-Pathway algorithms are 6.90, 7.36, 7.31, 7.31 seconds, respectively, and the average value of the time of this paper’s DDPG algorithm is 4.33 seconds, which is faster than the other renderer algorithms by 4.33 seconds, respectively, under the guarantee of generating the strokes with accuracy of 2.57, 3.03, 2.98, and 2.98 seconds. The data show that the DDPG algorithm proposed in this paper is able to quickly obtain oil painting creations that are more similar to the strokes of the content map with higher accuracy.

This subsection conducts a human perception study for the proposed model-based Deep Deterministic Gradient Policy Algorithm (DDPG) to further analyze its effectiveness. In that study, web pages of oil painting images generated by different models were presented to participants engaged in artistic and non-artistic fields, and the participants were asked to score or describe the quality perception of a single oil painting image. Feedback data from the participants were collected and statistically analyzed to derive an assessment of the advantages of the proposed model over the other models, including the visual quality of the generated oil painting images and the accuracy of the artistic style of the oil paintings. The score ranges from 1 to 5, with 5 being the most attractive. A total of 1200 ballots were collected from 30 different participants, and the distribution of scores and the average score for each model are shown in Table 2, where the rows represent the number of votes for the scores obtained and the last column represents the average score for each model. It can be seen that the DDPG model proposed in this paper obtained more high-scoring votes (votes greater than 3 points), with 53.58% of the total number of high-scoring votes and the highest average score (808.2). None of the other models’ high-scoring ballots accounted for more than 35% of the total ballots, and their mean scores were between [606, 707.8], which is a large difference from the DDPG model. These results indicate that the oil painting images generated using the DDPG model in this paper have a rich oil painting art style, which can maximize people’s needs and gain subjective recognition from viewers.

After completing the subjective evaluation, the effectiveness of the DDPG model proposed in this paper on oil painting creation generation is again verified through objective evaluation. In this subsection, quantitative experiments are first conducted based on the DDPG model, and the results show that the oil painting images generated by the DDPG method proposed in this paper can effectively train the corresponding scene image recognition model. In order to demonstrate the robustness of the DDPG method proposed in this paper, another experiment is conducted based on two additional scene recognition models. Then, qualitative experiments are added for comparison in terms of visual effect. In this paper, the oil paintings generated by the algorithm for each style type (classicism, romanticism, realism, impressionism, abstraction, etc.) are evaluated by using structural similarity (SSIM), mean square error (MSE), peak signal-to-noise ratio (PSNR), and perceived similarity (LPIPS) as the evaluation metrics. Finally, an objective evaluation was conducted to obtain a richer presentation of the results. The results of quantitative experiments are shown in Table 3.

Observing Table 3, it can be seen that the DDPG method achieves the best results in assisting the generation of five styles of oil paintings, namely, classical, romantic, realist, impressionist, and abstract, with SSIM, MSE, PSNR, and LPIPS values of 0.60, 0.0182, 12.80, and 0.3228, respectively, which exceed the results of other oil painting generation algorithms, and achieves the best objective evaluation.