Intelligent Recognition of Colour and Contour from Ancient Chinese Embroidery Images

Ancient Chinese embroidery works are collected in different museums, and the illuminant of each museum is different. Therefore, in order to enable designers to directly extract the colour and recognition of images taken during a visit to the museum, we do not use a special lightning cabinet with set illuminant. Only embroidery images taken in a non-dark environment can be recognised.

In this study, for the acquisition of ancient Chinese embroidery images, a combination of digital cameras were used [10], as well as a flat-panel scanner [9] to capture images. A Canon EOS REBEL T3i digital camera was used to capture ancient Chinese embroidery images, with the colour space of RGB, photometric mode of 5, P mode, and ISO AUTO. Under the lighting environment in the exhibition hall and without the glass barrier of the display cabinet, local embroidery patterns were directly shot from a close distance. Canon LIDE 300 flatbed scanners at a 2400DPI scan resolution, in the photo scan mode, and 300 DPI were applied.

The flatbed scanners can scan small ornaments in an A4 paper size range, and clothing larger than this size was taken with a Canon EOS REBEL T3i. In this study, because the actual size of the image taken and scanned was large, the cubic interpolation method based on a 4×4 pixel neighborhood was used to compress the images to shorten the time of computer operation.

Finally, we selected 80 embroidery samples, including three themes: plants, animals and characters, and selected representative experimental results from the three themes appearing in the illustrations of this study during the experiment.

Step1: Adjustment of colour levels.

In the experiment, since some old objects and the dim pattern colours would affect the discrimination of the pattern contour, the step of colour level adjustment was added in the pretreatment. This is one of the innovations of the method proposed in this study. The effect was to overcome the colour concentration and increase the difference between colours. This treatment achieved visual beauty, and improved the calculation gradient in contour extraction.

Because the black field and white field in the histogram [19] of 40 images in the colour experiment are different, the black field does not start from 0 and the white field does not end at 255. Therefore, it is necessary to move the two fields of each image to the corresponding starting and ending positions. In the processing, the total number of pixels is counted according to the size of each image, then the number of pixels is accumulated in turn by the means of following the direction in which the grey value increases from zero.

When the number of times is greater than the threshold, the pixel point is given a new grey value of 0. The same method is applied to accumulate the gray value of the white field, starting following the direction in which the gray value decreases from zero. Two relevant equations, 1 and 2, are presented: (1) sum1+=hist(i)s.t.sum1≥η×w×hminlevel=i \matrix{{{\rm{su}}{{\rm{m}}_1} + = {\rm{hist}}({\rm{i}})} \cr {{\rm{s}}.{\rm{t}}.{\rm{su}}{{\rm{m}}_1} \ge \eta \times w \times h} \cr {{\rm{minlevel}} = {\rm{i}}} \cr} (2) sum2+=hist(256−i)s.t.sum2≥η×w×hmaxlevel=256−i \matrix{{{\rm{su}}{{\rm{m}}_2} + = {\rm{hist}}(256 - {\rm{i}})} \cr {{\rm{s}}.{\rm{t}}.{\rm{su}}{{\rm{m}}_2} \ge \eta \times w \times h} \cr {{\rm{maxlevel}} = 256 - {\rm{i}}} \cr}

Where i ranges from 0 to 255; sum₁ is an intermediate quantity or count of the number of times the gray value appears; hist(i) is the number of times the gray value appears in i; η is the set threshold (=0.01); w and h represent the width and height of the picture, respectively, and minlevel and maxlevel represent the gray values of the new white and black fields in the original image, respectively. Finally, image A is obtained shown in Figure 4. In our experiment, each image to be manipulated is called the original image, which will be called image A after the adjustment of colour levels, as shown in the figure below, and image A is an epithet which exists in the form of a three-dimensional matrix in the processing.

Fig. 4

Step 2: Using Gaussian filtering to reduce noise.

This step traverses each pixel point (x, y) in the image with a 3×3 Gaussian mask of an integer shape. Equation 3 is as follows: (3) G(x,y)=12πσ2e−x2+y22σ2 {\rm{G}}\left({{\rm{x}},{\rm{y}}} \right) = {1 \over {2\pi {\sigma ^2}}}{{\rm{e}}^{- {{{{\rm{x}}^2} + {{\rm{y}}^2}} \over {2{\sigma ^2}}}}}

Where sigma is Gaussian standard deviation, which has a certain influence on the effect of filtering. The coordinates of the relative centre point (x, y) in the Gaussian mask are represented in the equation. In the process of traversal, the noise in image A from Step 1 is filtered out to form a smoother image A' (Figure 5b). Equation 4 is as follows: (4) A'=[121242121]*A {{\rm{A}}^{'}} = \left[{\matrix{1 & 2 & 1 \cr 2 & 4 & 2 \cr 1 & 2 & 1 \cr}} \right]*{\rm{A}}

Fig. 5

Regardless of the image, the form of filter is the same, because in other processing, it has an adaptive function for different images.

Step 3: Image sharpening.

Because of the characteristics of embroidery and the influence of photography, some transitional pixels are produced between the pattern and the background, which constitute a large number of gradual colours, resulting in the extraction of useless colours from colour recognition. In the experiment, yellow and light red are the most obvious on the black bottom plate. Because the edge of the image is too large, the number of gradual discolouration is large as well, and finally the colour that does not exist in the image will appear during colour recognition. Therefore, we used a sharpening mask to minimise the transition pixel points as much as possible to compensate for the image contour, enhance the image edge and make the edge clearer.

Image A' obtained from the previous step is sharpened to form a new image B (Figure 5c). Equation 5 is as follows: (5) B=[0−10−15−10−10]*A' {\rm{B}} = \left[{\matrix{0 & {- 1} & 0 \cr {- 1} & 5 & {- 1} \cr 0 & {- 1} & 0 \cr}} \right]*{{\rm{A}}^{'}}

Step 4: Colour smoothing.

For knitted embroidery patterns, when only Gauss filtering is used to remove noise, the pattern still has the embroidery thread inside the staggered texture. But in fact, they belong to a whole, which is detrimental in both colour extraction and contour extraction. Hence, it is also necessary to neutralize the similar colour by smoothing the colour level, thus reducing the extraction error.

A 5-dimensional iteration space (x, y) is created on physical coordinates and the 3D vectors of any colour space (e.g. the blue value of the RGB space, Lab space three channel values). Each time, a vector that satisfies the constraints which is in the 5-dimensional iteration space is added to form an offset vector, which constantly updates the coordinates of points until the coordinate change of the point to be updated is ignored. The final results are shown in Figure 5d. The points satisfying the following conditions in this 5-dimensional space are the basis of the synthesis vector of the update point. Equation 6 is as follows: (6) X−f≤x≤X+fY−f≤y≤Y+f‖(R,G,B)−(r,g,b)‖≤c \matrix{{{\rm{X}} - {\rm{f}} \le {\rm{x}} \le {\rm{X}} + {\rm{f}}} \cr {{\rm{Y}} - {\rm{f}} \le {\rm{y}} \le {\rm{Y}} + {\rm{f}}} \cr {\left\| {\left({{\rm{R}},{\rm{G}},{\rm{B}}} \right) - \left({{\rm{r}},{\rm{g}},{\rm{b}}} \right)} \right\| \le {\rm{c}}} \cr}

Where f and c are the setting values of the physical space range and the colour space range, respectively; x, y and r, g, b are the physical coordinates and colour 3D vector of the point to be updated, respectively; X,Y are the physical coordinates that satisfy the surrounding constraints, and R,G,B are a three-dimensional vector of colour around which the limiting conditions are satisfied.

Objects could be clustered by their colour [20]; K-means is one of the classical algorithms in clustering [21], which is applied to separate the image background and edge [22]. Compared with K-means, we only need to select a sample point randomly as the initial cluster center C1 when we use K-means++ [23]. First, the shortest distance between each sample and the current existing cluster center, is calculated by D(x). Then the probability that each sample point is selected as the next cluster center is calculated. Equation 7 is as follows: (7) D(x)2∑i=1nD(xi)2 {{{\rm{D}}{{\left({\rm{x}} \right)}^2}} \over {\sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {D{{\left({{{\rm{x}}_{\rm{i}}}} \right)}^2}}}}

Finally, according to the roulette method, the next cluster center is selected. The sample point farther away from the existing cluster center point is easier to be selected and is more consistent with the difference between classes. The roulette method is repeated until the cluster center equal to the number of K values is selected.

The next operation is the same as the iterative principle of K-means. Assuming that the cluster (C_i) is divided into (C₁, C₂,……C_k), our goal is to minimise the square error E (Equation 8): (8) E=∑i=1k∑x∈Ci‖x−μi‖22 {\rm{E}} = \sum\nolimits_{{\rm{i}} = 1}^{\rm{k}} {\sum\nolimits_{{\rm{x}} \in {{\rm{C}}_{\rm{i}}}} {\left\| {{\rm{x}} - {\mu _{\rm{i}}}} \right\|_2^2}}

Where x is the 3D vector of RGB values in the picture, and μ_i is the (C_i) mean vector. μ_i is computed as follows (Equation 9): (9) μi=1|Ci|∑x∈Cix {\mu _{\rm{i}}} = {1 \over {\left| {{{\rm{C}}_{\rm{i}}}} \right|}}\sum\nolimits_{{\rm{x}} \in {{\rm{C}}_{\rm{i}}}} {\rm{x}}

K-means++ clustering is more consistent with the distribution of sample data than K-means. The intraclass similarity and interclass differentiation are both higher. We need to set the K value according to the number of colours in the selected image. If there are 12 colours with an obvious colour difference in an image, we need to set K=12. After the operation, 12 colours can be identified and arranged from more to less according to the number of pixels of the same colour (Figure 6).

Fig. 6

Canny first proposed a calculation method of image edge detection using the Canny operator [24]. Experiments in recent years show that the best segmentation effect for a printed fabric pattern is to use the Canny operator [25]. Hence, the Canny operator is chosen to detect the edge profile of an embroidery pattern. The methodology consists of three steps.

Step1: Two sets of 5×5 Sobel operators are used to traverse the image from the X-axis and Y-axis, respectively, and the preliminary contour is extracted. Equation 10 is as follows: (10) Gx=[−1−2021−4−8084−6−120122−4−8084−1−2021]*AandGy=[1464128128200000−4−8−12−8−4−1−4−6−4−1]*A \matrix{{{{\rm{G}}_{\rm{x}}} = \left[{\matrix{{- 1} & {- 2} & 0 & 2 & 1 \cr {- 4} & {- 8} & 0 & 8 & 4 \cr {- 6} & {- 12} & 0 & {12} & 2 \cr {- 4} & {- 8} & 0 & 8 & 4 \cr {- 1} & {- 2} & 0 & 2 & 1 \cr}} \right]*{\rm{A}}} \cr {{\rm{and}}} \cr {{{\rm{G}}_{\rm{y}}} = \left[{\matrix{1 & 4 & 6 & 4 & 1 \cr 2 & 8 & {12} & 8 & 2 \cr 0 & 0 & 0 & 0 & 0 \cr {- 4} & {- 8} & {- 12} & {- 8} & {- 4} \cr {- 1} & {- 4} & {- 6} & {- 4} & {- 1} \cr}} \right]*{\rm{A}}} \cr}

Where G_x and G_y represent images of the X-axis and Y-axis, respectively, and A is the image processed by the Gaussian filter.

Step 2: The amplitude matrix G (Equation 11) and angle matrix θ of the image (Equation 12) are calculated to suppress the non-maximum value: (11) G=Gx2+Gy2 {\rm{G}} = \sqrt {{\rm{G}}_{\rm{x}}^2 + {\rm{G}}_{\rm{y}}^2} (12) θ=atan2(Gy,Gx) \theta = {\rm{atan}}2\left({{{\rm{G}}_{\rm{y}}},{{\rm{G}}_{\rm{x}}}} \right)

By comparing the size of the corresponding domain values in the 3×3 domain, the pixel points and the two adjacent pixel points at the corresponding angle are compared to eliminate the non-maximum value (Figure 7).

Step 3: With a double threshold algorithm, a high threshold (1200) and low threshold (200) are set to make the contour more accurate during the experiment. The gradient amplitude higher than 1200 belongs to the profile, and the gradient amplitude lower than 200 is excluded. The gradient amplitude between the two is needed to be judged. That connected with the determined contour belongs to the contour, and that not connected with the determined contour should be excluded. The final results are shown in Figure 8.

Fig. 7

Fig. 8

Forty images were randomly selected from 80 images to identify colour for the first time. First, the Gaussian filter and smooth colour were used for preprocessing, and then K-means++ clustering was carried out, which cannot correctly identify all colours. Secondly, the steps of pretreatment were improved. The adjusted colour levels were added before Gaussian filtering, and then K-means++ clustering was carried out. The result of colour identification is shown in Figure 12b.

Fig. 12

When the grey value in the histogram is relatively concentrated, the colour is recognized by K-means++ clustering, and the resultant saturation is low, and the sense of colour to the designer is weak when used as the design material. After we move the black field and the white field to adjust the colour levels, we can improve the colour saturation, identify more bright colours, and better use them in the design.

Sharpening was added after Gaussian filtering in the third experiment, and then K-means++ clustering was carried out (Figure 12c). Thus, after adding ‘sharpening’, the recognized colour is closer to the original image both in hue and quantity, especially for the image with a large difference between the target and background colours.

The same 40 images were used in the contour extraction experiment. In the first experiment, the Canny operator was adopted to extract contour directly after Gaussian filtering (Figure 13). In the second experiment, the step of colour level adjustment was added before Gaussian filtering (Figure 13c). Results show the edge lines detected in the second experiment are more coherent, and fewer meaningless points and lines are detected, which are more consistent with the contour edges of the original image (Figure 13d). When the grey value in the histogram is relatively concentrated, the change between adjacent pixels is not very obvious, which will affect the recognition of the image contour. After the black field and the white field are moved to adjust the colour levels, the original grey value is stretched, the concentrated grey value scattered, and the grey value difference between adjacent pixels is increased, which makes contour identification easier.

Fig. 13

Because the image acquisition was from ancient Chinese embroidery, a part of the physical colours are dim due to age or the influence of environmental illumination, which slightly affect the hue and saturation and thus are not conducive to colour and contour extraction. After the above adjustment, the colour and edge of the image can be optimised, which is more conducive to colour and contour extraction.

Colour extraction experiments from multiple embroidery images in RGB, HSV and LAB colour spaces were conducted using K-means++ clustering. Results show the RGB space is more suitable for the colour extraction of ancient Chinese embroidery images (Figure 14).

Fig. 14

Thus, the use of the HSV space is prone to obvious errors, while the use of LAB and RGB spaces leads to small errors and similar results. However, the use of the LAB space requires two colour conversions, while that of the RGB space with batch image processing avoids colour conversion and, hence, returns results faster. Thus, we chose the RGB space in later experiments.

K-means++ clustering parameters were set according to the number of colours in the image and thus were different among images. After many experiments, we tried to combine a variety of high and low thresholds and finally selected 1200 and 200, respectively, which are more suitable for the calculation of embroidery images. Contour extraction after the threshold setting can avoid the extraction of redundant contours (Figure 15).

Fig. 15