Oil Painting Stylisation of Multi-scale Visually Layered Images Based on Drawing Algorithms

In recent years, computer algorithms have made significant progress in non-realistic painting (NPR) techniques such as oil painting, watercolour painting, and dot painting, and the practical applications of these techniques have received increasing attention (Hertzmann, A, 1998)[1]. Existing NPR methods can be broadly classified into two categories: one simulates the raw material for painting through physical models; the other focuses on simulating the creative process of real artists, generating stroke shapes layer by layer through optimisation or greedy algorithms to achieve the rendering of painting effects.

Image stylisation is an important research direction in Non-Realistic Rendering (NPR), which is able to show unique artistic styles such as watercolour, ink, and oil painting while preserving the main visual features of the image. Oil painting is one of the most representative styles due to its long history, deep cultural heritage and rich expressive power. Creating a high-quality oil painting requires an organic combination of painting tools, techniques and experience (Hertzmann, 2001; Hays & Essa, 2004))[2-3]. Therefore, the use of computers to simulate the creation of oil paintings and generate images with oil painting style has become a popular research topic in the field of image stylisation.

In order to achieve a realistic oil painting effect, traditional oil painting stylisation methods mostly generate oil painting effects by simulating the size, direction and other characteristics of oil painting brushes. Other methods use texture synthesis techniques to extract texture from the reference oil painting image and apply it to the target image to achieve the oil painting stylisation effect. In addition, some methods use fluid simulation to model the interaction between brushes, oils and canvas to achieve highly realistic details. In recent years, deep learning has made significant progress in areas such as computer vision and speech processing, and has been gradually introduced into the study of image stylisation (Hertzmann, 2003; Johnson, Alahi, & Fei-Fei, 2016; Gatys, Ecker, & Bethge, 2016)[4-6]. Deep learning-based oil painting stylisation methods usually use end-to-end neural networks to generate oil painting stylisation effects by directly processing input images.

In the image processing process of oil painting stylisation, there are several key issues to be solved. First, How to create a reference image hierarchy that aligns better with the painting process.. The unique charm of oil paintings lies in the layering of their representations of scenes, and painters usually use the same colours and brushstrokes to gradually present the details of each layer (Semmo, A., et al., 2018; Colburn, A., et al., 2014; Hertzmann, A., et al., 1999)[7-9]. Therefore, it is necessary to establish a layered model based on the input image, and gradually express the details through a coarse-to-fine approach. Secondly, How to create an efficient oil painting brushstroke model. In the drawing process, artists convey visual and psychological intentions through factors like stroke size, starting point, and direction. In computer drawing, the strategy of arranging these strokes is crucial to the visual effect, and traditional oil painting stylisation algorithms tend to ignore this complex drawing process and fail to accurately express local details and themes. Although existing algorithms have made progress in oil painting stylisation and achieved certain results, most of them focus on pixel-level uniform processing and do not fully simulate the actual painting steps of the artist. In order to better reproduce the creative process of oil painting, this paper proposes a hierarchical oil painting stylisation algorithm based on multi-scale brushes, aiming to simulate the artist's painting style and enhance the stylisation effect.

The stroke-based oil painting stylization algorithm, first introduced by Hertzmann (Hertzmann, 2001,) [2], involves creating a multi-layer Gaussian pyramid reference image from the input image. Filters like Sobel are used to extract gradient information, which helps determine the stroke's starting point and direction, allowing the image to be drawn layer by layer. Specifically, the reference image sequence is processed through Gaussian filtering to create layers matching the original image's size. The stroke's starting point is identified using a grid that corresponds to the stroke radius, and pixel errors within each grid are analyzed to decide if a starting point should be generated. If the error surpasses a threshold, the pixel with the largest error is selected as the starting point. The stroke's direction is then defined by the gradient at this point. However, this approach only applies Gaussian filtering when building the hierarchical image sequence, does not consider the painter's perspective in terms of image decomposition, and overlooks the relationship between primary and secondary content when organizing strokes.

To enhance the realism of the oil painting effect, Hertzmann built upon his previous work (Hertzmann, A., et al., 1999)[9] and proposed a method based on a light model. The key concept is to first calculate the image's height field information, which is then combined with the Phong lighting model to simulate lighting effects, creating highlight and shadow areas to improve visual realism. Additionally, factors like edge transparency are incorporated into the stroke model to make the result more akin to a hand-painted oil painting style.

In order to solve the problem of over-simplified calculation of stroke direction in traditional algorithms, (Wei & Levoy. 2000)^[10]used radial basis function (RBF) to train the edge information weights in the strong edge region, and then generated the stroke direction of other regions by interpolation method. This method is suitable for the drawing of large textures, but is less effective when dealing with complex and texture-rich images.

In addition, literature (Paris & Durand. 2019)[11] proposes a learning-based stylisation algorithm, which first establishes a style transformation model between the input image and the stylised effect image, and then stylises another input image by an optimisation method. This method requires preparing the original image for learning and the stylised image first. Literature (Viola, P., Jones, M, 2001)[12], on the other hand, extracts stroke models from oil paintings, combines them with image segmentation techniques, applies the models to different regions, and finally uses texture synthesis to generate the stylised image.

Aiming at the problem of insufficiently fine calculation of stroke direction in the literature (Cabral & Leedom. 1993)[13], Hertzmann(Hertzmann, A., et al., 1999)[9] proposed an improved method: the key drawing region is determined through manual interaction, and the energy function combined with the relaxation iteration is used to achieve the rough drawing of the region. However, this method suffers from the disadvantage of slow computation speed and requires manual designation of the important region. In order to increase the diversity of brushstroke drawing, literature (Marr, D., Hildreth, E. C, 1980)[14] uses models of 3D geometric shapes such as small cubes for drawing.

This paper improves upon the traditional stroke-based oil painting algorithm. Unlike the global blurring effect of the traditional Gaussian pyramid method, a pyramid reference image sequence is created by combining bilateral filtering to address texture anisotropy, along with mathematical morphological operations. Additionally, for regions sensitive to the human eye, the paper enhances stroke boundary constraints and further optimizes the calculation of the starting point detection threshold and stroke direction at the finest layer.

In this paper, we propose a multiscale brush-based stylised painting algorithm for layered oil paintings, which simulates the actual painting process from coarse to fine. Based on a given brush size, the painting process is divided into multiple levels. In each level of painting, the algorithm combines the information of the target image and its tangent direction field, the content of the canvas that has been painted, and the brush size and shape of the current layer to determine the specific way of painting each stroke. To measure the pixel difference between the canvas and the target image, the L₂ paradigm of RGB colours is used in this paper. Based on the generated brush streamlines, the algorithm uses texture mapping technique (Johnson, Alahi, & Li. 2016)[5] to implement brush painting on the canvas.

The steps of the algorithm in this paper are as follows.

Inputs. Target Image I, Tangent Direction Field F, Previous Layer Canvas C_k–1, Incremental Voronoi Sequence S, Brush B_k.

Output. Current layer canvas C_k.

Step 1. Initialise the current layer canvas C_k with C_k–1.

Step 2. Calculate the pixel difference between C_k and the target image I, and set the sample point counter i=0.

Step 3. Compare i with length(S).

Step 3.1. If i < length(S), do next step; otherwise, go to Step4.

Step 3.2.Get the initial position of the brush line, and calculate the tangent direction field and colour value of the position; meanwhile, define the vertex array L=[0] of the brush line and increase i by 1;

Step 3.3.Compare i with length(S):

Step 3.4. If i < length(S), execute Step 4; otherwise, go to Step 4.

Step 4. Compare the length of L with the longest length of B_k.

Step3.3.1. If the length of L is less than the length of B_k, execute the next step; otherwise, go to Step3.4; Step3.3.2.

Step 3.3.2. Determine the current position: the error between the canvas and the target image pixel, and the error between the initial position and the target image pixel. Determine whether the error between the canvas and the target image, the difference in colour from the initial position and the difference in direction from the previous position exceed the given threshold. If the threshold is not exceeded, execute the next step; otherwise, go to Step3.4; Step3.3.3.

Step 3.3.3. Determine if the current position has been covered in the current layer. If it is not covered, do the next step; otherwise, go to Step3.4; Step3.3.4.

Step 3.3.4. Add the current position to L, and determine the next position of the brush streamline based on the tangent direction field of the position and the brush size B_k. At the same time, update the current position and its direction field, go to Step3.3; Step3.4.

Step 3.4. Compare the length of L with the shortest length of Bk. If the length of L is greater than the shortest length of Bk, perform the next step; otherwise, go to Step3; Step3.5.

Step 3.5. Draw the brush flow according to L and use the texture mapping method to draw the brush. At the same time, update the error between the canvas and the target image according to the painted canvas, go to Step3.

Step 5. Output the current layer canvas Ck.

In this paper, we use ordinary true-colour 32-colour JPEG compressed images as inputs, and the original inputs are shown in Fig. 1, which are stylized using both the traditional stroke-based oil painting algorithm and the improved algorithm to evaluate the enhancement in painting effects.

Figure 1.

Original Input Image.

As shown in Figure 2, compared to the traditional algorithm and the one proposed in this paper, the improved method produces a consistent blurring effect for large image blocks, aligning well with visual perception classification. Meanwhile, it retains the approximate texture of the image more clearly despite overall blurring. These improvements are attributed to the use of bilateral filtering on the reference layer. Additionally, secondary perceptual texture details in large pixel blocks are effectively removed, while sensitive small pixel blocks (e.g., the arm region) are preserved, which aligns with the theoretical analysis of the mathematical morphology operation. Figure 3 illustrates the boundary information obtained using the DoG method, which enhances the starting point and direction of the strokes based on the importance distribution in the reference map.

Figure 2.

Intermediate results of the Oil Painting Stylised Drawing Section

Figure 3.

Boundary Information Obtained from the Importance Distribution of the Reference Map

As shown in Figure 3, the drawing results of the proposed algorithm feature smoother and more natural strokes along the boundary. Additionally, the refinement of the finest layer results in a more detailed and realistic portrayal of the face, with improved accuracy in important areas like the human face. The enhanced stroke direction processing allows for better alignment with the texture direction, thanks to overall improvements in stroke direction. This leads to peripheral strokes better following the main boundary, as well as improved visual effects due to the optimized reference image generation and the boundary constraints applied to the importance regions.

As shown in Figure 4, the drawing result of this algorithm features smoother and more natural boundary strokes. The face is depicted with more detail and realism due to improvements in the finest layer, and the details in key areas, such as the face, are more accurate. The enhanced stroke direction processing better aligns with the texture direction, thanks to overall improvements in stroke orientation. which leads to the better following of the main boundary in the peripheral region, and also due to the improvement of the reference image generation algorithm and the improvement of the visual effect brought by the constraints of the important regions on the drawing of the boundary of the strokes.

Figure 4.

Image Oil Painting Stylised Drawing Results

This paper proposes a multi-scale layered image oil painting stylisation method with a layered oil painting drawing algorithm. The method simulates the real painting process from coarse to fine by using brushes of varying scales and painting layer by layer, from large to small. In each layer, the brush position is first determined using an incremental Voronoi sequence, and the tangent direction field is calculated with the structure tensor to set the brush flow direction. Additionally, the algorithm constrains features such as brush length and curvature to make the streamlines appear more natural and realistic. By combining these streamlines with the brush shape and height field, the effect of a real oil painting is simulated through texture mapping. Experimental results demonstrate that the algorithm effectively replicates the painting process and produces high-quality oil painting style images. For the shortcomings of the algorithm, further research and improvement will be carried out subsequently.