1. bookAHEAD OF PRINT
Dettagli della rivista
License
Formato
Rivista
eISSN
2444-8656
Prima pubblicazione
01 Jan 2016
Frequenza di pubblicazione
2 volte all'anno
Lingue
Inglese
Accesso libero

Garment Image Retrieval based on Grab Cut Auto Segmentation and Dominate Color Method

Pubblicato online: 15 Jul 2022
Volume & Edizione: AHEAD OF PRINT
Pagine: -
Ricevuto: 08 Feb 2022
Accettato: 07 Apr 2022
Dettagli della rivista
License
Formato
Rivista
eISSN
2444-8656
Prima pubblicazione
01 Jan 2016
Frequenza di pubblicazione
2 volte all'anno
Lingue
Inglese
Introduction

The rapid development of E-commerce and online shopping platforms, such as the most popular ones of Taobao.com, JD.com, and VIPshop.com, have attracted more and more consumers to purchase clothes online. As a result, images on the network have seen explosive growth. However, these platforms mainly require users to retrieve products by text, which means that consumers need to describe their wanted clothes in text. If they fail to describe the product clearly, they will get numerous non-related results, which is a waste of time browsing and likely finding no fine clothes. On the other hand, the text-based retrieval method requires shop owners to upload garment images, mark manually, and describe clothes features like colors, forming a text basis for retrieval. This actually costs a lot of time and labor fees and can no longer satisfy the demands at present. Due to the above, how to rapidly and correctly find what consumers want from numerous garment images is a problem to be solved urgently.

Although scholars from both home and abroad have conducted painstaking work on the segmentation of garment images and the extraction of garment image color features, the algorithms to segment images and extract image color features need to be further improved due to the complexity of the work. Xie Ying proposed an image segmentation technique that combines the Mean Shift algorithm and the K means clustering algorithm. It reduces the computation amount while improving the segmentation effect of the garment image[1]. Guo Xinpeng proposed a bisection segmentation method based on a priori knowledge, which uses block truncation encoding to truncate the three-dimensional color space to a six-dimensional space and achieve image segmentation according to the appearing laws of object and background[2]. Han Dong Yue et al. proposed a watershed merger segmentation algorithm that roughly segments images according to the color gradients and then achieves accurate segmentation combining with the merger algorithm for the manually marked region[3]. Luo Min et al. combined the traditional C mean fuzzy clustering algorithm with the Gaussian kernel function and utilized the local spatial information to improve image resolution, which improved the segmentation effect of the fuzzy clustering algorithm to a large extent[4]. Chen Qian et al. proposed an improved color feature extraction algorithm based on the weighted Euclidean distance, weighted the segmentation image histogram algorithm, the foreground color histogram, and the color moment algorithm[5]. Tao Binjiao et al. obtained the optimal block size and the weighted ratio through experimental comparison. They obtained a new color feature extraction algorithm combining with Grab Cut segmentation[6]. Li Jian et al. used a balanced histogram to enhance the spatial color pixel when conducting the color space conversion of the image. After the conversion, they used the K-Means clustering algorithm to enhance the image again, thereby enhancing the efficiency of color feature extraction[7]. Hou Yuanyuan et al. proposed a garment image retrieval method that combines the ResNet50 deep feature and the traditional color feature took full use of the effectiveness and hierarchal of the residual network when extracting image features and combined the deep features and color features of the garment image as the final feature vector, and then performed similarity measurement[8].

The garment image retrieval method proposed in this study, based on Grab Cut auto segmentation and dominate color method, can rapidly and effectively retrieve garment images.

Foreground extraction of garment images based on Grab Cut auto segmentation
Grab Cut segmentation

The basic idea of Grab Cut is to map the entire garment image that is to be processed into an S-T network. The S-T network consists of vertices and edges, where: S is a source point, representing the pixel point of the image foreground; T is a sink, representing the pixel point of the image background, and the edges formed with various pixels. Whether these pixel points belong to the foreground or background is judged by these edges. After obtaining the foreground information by this approach, the max. Flow or the min. A segmentation algorithm would be used to achieve image segmentation based on the established energy function.

ROI region selection

ROI stands for the region of interest, namely the foreground region. It is pretty crucial in Grab Cut image segmentation algorithm. The selection of every point would affect the results of the segmentation. Improper selection would lead to excessive or insufficient image segmentation. Manual selection of points profoundly relies on the operator's experience, which costs much time. However, automatic selection of ROI is also available, which is more convenient and efficient.

Gaussian model establishment

After well selecting the ROI region, to obtain minimal Gibbs energy and achieve the purpose of segmentation, it needs to perform modeling for image foreground and background by the Gaussian model before algorithm iteration. Moreover, for each time of iteration, optimal GMM parameters must be set.

Gibbs energy function used in segmentation is: F(α,k,θ,z)=G(α,k,θ,z)+R(α,z) {\rm{F}}\left( {\alpha ,{\rm{k}},\theta ,z} \right) = {\rm{G}}\left( {\alpha ,{\rm{k}},\theta ,{\rm{z}}} \right) + {\rm{R}}\left( {\alpha ,{\rm{z}}} \right) Where F is Gibbs energy; G is the data item; R is the smooth item

The Grab Cut auto segmentation algorithm is used to extract the foreground effects of garment images with a simple or complex background, see Figures 1 and 2.

Figure 1

Grab Cut auto segmentation of garment image with a simple background

Figure 2

Grab Cut auto segmentation of garment image with complex background

It can be seen from Figures 1 and 2 that the Grab Cut auto segmentation algorithm shows satisfactory performance in extracting the foreground of the garment images with simple and complex background. Therefore it is applied in this study to process the garment images, preparing for the following step computation.

Color feature extraction and similarity measurement
Color feature expression
Color coherence vector

The color coherence vector can express the spatial distribution of image color. It divides each color group of the color histogram into two parts according to the spatial relations among pixels: cohesive and non-cohesive parts. Suppose that the given threshold is V; if specific pixels of either part occupy an area more significant than the given threshold V, they would be defined as cohesive pixels. Otherwise, pixels shall be defined as non-cohesive pixels.

The histogram of an image has multiple groups of colors. The cohesive pixels and non-cohesive pixels of the first group of colors are represented by α1 and β1 respectively, and so on, then the i-th would be αi and βi. The color coherence vector of this image (CCV) would be expressed as below: <(α1,β1),(α2,β2),,(αi,βi)> < \left( {{\alpha _1},\,{\beta _1}} \right),\,\left( {{\alpha _2},\,{\beta _2}} \right),\, \cdots ,\,\left( {{\alpha _{\rm{i}}},\,{\beta _{\rm{i}}}} \right) >

The histogram of this image can be expressed as: <α1+β1,α2+β2,αi+βi> < {\alpha _1} + \,{\beta _1},\,{\alpha _2} + \,{\beta _2}\, \cdots ,\,\,{\alpha _{\rm{i}}} + \,{\beta _{\rm{i}}} >

Step1: Convert RGB color space to HSV color space.

Step2: Quantify color by the uniform quantization method.

Step3: Divide related areas.

Step4: Judge the coherence property, that is, to judge cohesive or non-cohesive pixels by the set threshold

Dominate color method

When picking clothes, people are usually attracted by the dominant color of the clothes first before notifying other details. The dominant color actually refers to the color accounts most among all colors on the clothes. Therefore, it is pretty essential to extract and describe the dominant color of the clothes. As for the extraction, we first divide into several segments, quantify through three channels of the color space, and then sort out according to the total pixels of each color. Each composing part of the color space is further divided into several parts, either a uniform or non-uniform. After that, we scan images and classify color components according to pixels; calculate the total number of pixels of each color; and sort out color types according to their total pixels; and define the first m types as the dominant colors.

Step1: Convert RGB color space to HSV color space.

Step2: Define a zero matrix with four rows and 72 columns and store values of three components of H, S, V values to No.2, 3, and 4 rows of each column.

Step3: Divide the H component into eight parts, and divide the S and V components into three parts each.

Step4: Take the color components of the center of H, S, and V after the previous segmentation.

Step5: Take the first eight colors as the dominant colors.

Similarity measurement

After selecting a garment image and extracting its color features, we face the problem of retrieving multiple images similar to or the same to this image. To solve this problem, we should compare this image with images in the library. The similarity measurement: matching features with those in the image library and outputting results according to the degree of similarity. Minkowsky distance is defined based on LP norm and expressed as: Lp(M,N)=[i=1n|aibi|p]1p {{\rm{L}}_{\rm{p}}}\left( {{\rm{M}},{\rm{N}}} \right) = {\left[ {\sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {{{\left| {{{\rm{a}}_{\rm{i}}} - {{\rm{b}}_{\rm{i}}}} \right|}^{\rm{p}}}} } \right]^{{1 \over {\rm{p}}}}} When p=1, L1 (M, N) are the block distance L1(M,N)=i=1n|aibi| {{\rm{L}}_{\rm{1}}}\left( {{\rm{M}},{\rm{N}}} \right) = \sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {\left| {{{\rm{a}}_{\rm{i}}} - {{\rm{b}}_{\rm{i}}}} \right|} When p=2, L2 (M, N) are Euclidean distance L2(M,N)=(i=1n(aibi)2)12 {{\rm{L}}_{\rm{2}}}\left( {{\rm{M}},{\rm{N}}} \right) = {\left( {\sum\nolimits_{{\rm{i}} = 1}^{\rm{n}} {{{\left( {{{\rm{a}}_{\rm{i}}} - {{\rm{b}}_{\rm{i}}}} \right)}^{\rm{2}}}} } \right)^{{1 \over {\rm{2}}}}} When p=∞, L (M, N) are Chebchv distance L1(M,N)=maxi=1|aibi| {{\rm{L}}_{{\rm{1}}\infty }}\left( {{\rm{M}},{\rm{N}}} \right) = \mathop {{\rm{max}}}\limits_{{\rm{i}} = 1} \left| {{{\rm{a}}_{\rm{i}}} - {{\rm{b}}_{\rm{i}}}} \right| Where (M, N) is the feature vector of the image, n is the feature vector dimensions of the image. The Euclidean distance is used in this study, which is featured in fast computing speed and is applicable to the small-sized database of this study.

Recall and precision ratios

The precision and recall ratios are used in this paper to evaluate the retrieval results. Assuming that in retrieval, the image to be retrieved is I, then the recall ratio is the rate of all retrieved images similar to the i-image to all the images related to the I image in the image database. Moreover, the precision ratio is the proportion of all retrieved images similar to i-images in all retrieved images. The definition is as follows:

Definitions of precision and recall ratios

Related Non-related
Retrieved A (all related images retrieved) C (non-related images retrieved)
Not retrieved B (images not retrieved from the library) D (excluded non-related images)

Recallratio=AA+B {\rm{Recall}}\,{\rm{ratio}} = {{\rm{A}} \over {{\rm{A}} + {\rm{B}}}} , precisionratio=AA+C {\rm{precision}}\,{\rm{ratio}} = {{\rm{A}} \over {{\rm{A}} + {\rm{C}}}}

Experiment results and analysis

MATLAB language is used to design an image retrieval system in this study, which can efficiently retrieve the required images from the image library and engage in manual interaction to conduct many experiments, thereby verifying which algorithm has better retrieval efficiency. In this study, the color feature is extracted by the color coherence vector and the dominant color method to retrieve garment images and compare retrieval results. Considering a practical perspective, the 300 garment images in the image library are all downloaded from websites such as Tmall and JD.com. These pictures are divided into 50 groups, each containing six pictures. These six images are shot from different angles of the same clothes with different backgrounds. The output result of the system is set to be ten similar images, so the precision ratio of each image is up to 60%, while the highest recall ratio is 100%. Figures 3 and 4 show the garment image retrieval results obtained by taking anyone garment image, for instance, to extract color features by color coherence vector and dominate color method, respectively.

Figure 3

Retrieval results obtained by extracting color features with color coherence vector

Figure 4

Retrieval results obtained by extracting color features with the dominant color method

The two algorithms of color feature extraction are both used to retrieve every image of every group. The retrieved amounts of images are shown in Figure 5 to Figure 10.

Figure 5

Retrieval result comparison of the first image

Figure 6

Retrieval result comparison of the second image

Figure 7

Retrieval result comparison of the third image

Figure 8

Retrieval result comparison of the fourth image

Figure 9

Retrieval result comparison of the fifth image

Figure 10

Retrieval result comparison of the sixth image

It can be seen from Figure 5 to Figure 10 that in the 50 groups of images, no matter which garment image is used as the retrieval object, the output results of retrieval by extracting color feature with the dominant color method is usually more significant than the output results obtained by color feature extraction with color coherence vector.

Figures 11–12 are the average precision and recall rates calculated from using color coherence vectors to extract color features for garment image retrieval and results of using dominate color method to extract color features for garment image retrieval.

Figure 11

Comparison of precision ratios between color coherence vector and dominant color method

Figure 12

Comparison of recall ratios between color coherence vector and dominant color method

It can be seen from Figure 11 that the precision ratio of garment image retrieval by extracting color features with dominant color method is higher than that obtained by extracting color features with color coherence vector, reaching around 40%. Figure 12 shows that the recall ratio of garment image retrieval by extracting color features with the dominant color method is higher than that of garment image retrieval by extracting color features with the color coherence vector, reaching about 65%.

Conclusions

In this study, a garment image retrieval method based on Grab Cut auto segmentation, and dominant color method is proposed, extracting foreground of garment images by Grab Cut auto segmentation algorithm, and then conducting garment image retrieval by extracting color features with dominate color method. Through experimental tests on a sample library of 300 clothes, it is found that the Grab Cut auto segmentation algorithm has better results in extracting the foreground of garment images with simple and complex backgrounds. Compared with the color coherence vector algorithm, the dominant color method could improve garment image retrieval effects.

Figure 1

Grab Cut auto segmentation of garment image with a simple background
Grab Cut auto segmentation of garment image with a simple background

Figure 2

Grab Cut auto segmentation of garment image with complex background
Grab Cut auto segmentation of garment image with complex background

Figure 3

Retrieval results obtained by extracting color features with color coherence vector
Retrieval results obtained by extracting color features with color coherence vector

Figure 4

Retrieval results obtained by extracting color features with the dominant color method
Retrieval results obtained by extracting color features with the dominant color method

Figure 5

Retrieval result comparison of the first image
Retrieval result comparison of the first image

Figure 6

Retrieval result comparison of the second image
Retrieval result comparison of the second image

Figure 7

Retrieval result comparison of the third image
Retrieval result comparison of the third image

Figure 8

Retrieval result comparison of the fourth image
Retrieval result comparison of the fourth image

Figure 9

Retrieval result comparison of the fifth image
Retrieval result comparison of the fifth image

Figure 10

Retrieval result comparison of the sixth image
Retrieval result comparison of the sixth image

Figure 11

Comparison of precision ratios between color coherence vector and dominant color method
Comparison of precision ratios between color coherence vector and dominant color method

Figure 12

Comparison of recall ratios between color coherence vector and dominant color method
Comparison of recall ratios between color coherence vector and dominant color method

Definitions of precision and recall ratios

Related Non-related
Retrieved A (all related images retrieved) C (non-related images retrieved)
Not retrieved B (images not retrieved from the library) D (excluded non-related images)

S.W. Lee, S. Sarp, D.J. Jeon and J.H. Kim, Smart water grid: the future water management platform, Desal. Water Treat., 2015.55 (2):pp.339–346 LeeS.W. SarpS. JeonD.J. KimJ.H. Smart water grid: the future water management platform, Desal Water Treat 2015 55 2 339 346 10.1080/19443994.2014.917887 Search in Google Scholar

Xie Ying, Segmentation method combined with mean shift and K-mean clustering algorithm for clothing image. Electronic Measurement Technology., 2013.36(8):53–55 XieYing Segmentation method combined with mean shift and K-mean clustering algorithm for clothing image Electronic Measurement Technology 2013 36 8 53 55 Search in Google Scholar

GUO Xin-peng, HUANG Yuan-yuan, HU Zuo-jin, Unsupervised Complex-scene Clothing Image Segmentation Algorithm Based on Color and Texture Features. Computer Science., 2017.44(11A):229–232 GUOXin-peng HUANGYuan-yuan HUZuo-jin Unsupervised Complex-scene Clothing Image Segmentation Algorithm Based on Color and Texture Features Computer Science 2017 44 11A 229 232 Search in Google Scholar

HAN Dong-Yue, QIN Xiao, LIU Bu-shi, Color Clothing Image Segmentation Based on Region Merging of Watershed. Journal of Guangxi Teachers Education University(Natural Science Edition)., 2015.32(4):75–81 HANDong-Yue QINXiao LIUBu-shi Color Clothing Image Segmentation Based on Region Merging of Watershed Journal of Guangxi Teachers Education University(Natural Science Edition) 2015 32 4 75 81 Search in Google Scholar

LUO Min, LIU Dong-bo, WANG Ning, Clothing Image Segmentation Based on Improved Kernel Fuzzy C-means Clustering Algorithm. Journal of Hunan Institute of Engineering(Natural Science Edition)., 2018.28(2):40–43, 69 LUOMin LIUDong-bo WANGNing Clothing Image Segmentation Based on Improved Kernel Fuzzy C-means Clustering Algorithm Journal of Hunan Institute of Engineering(Natural Science Edition) 2018 28 2 40 43 69 Search in Google Scholar

CHEN Qian, PAN Zhong-liang, Research and improvement of color feature extraction algorithms in the content-based clothing image retrieval system. Laser Journal., 2016.37(4):62–68 CHENQian PANZhong-liang Research and improvement of color feature extraction algorithms in the content-based clothing image retrieval system Laser Journal 2016 37 4 62 68 Search in Google Scholar

TAO Binjiao, CHEN Qian, PAN Zhongliang, Improvement of block weighted color histogram algorithms in clothing image retrieval system. Laser Journal., 2017.38(8):97–101 TAOBinjiao CHENQian PANZhongliang Improvement of block weighted color histogram algorithms in clothing image retrieval system Laser Journal 2017 38 8 97 101 Search in Google Scholar

LI Jian, JIANG Nan, BAOYIN Bate, Spatial Color Clustering Algorithm and Its Application in Image Feature Extraction. Journal of Jilin University(Science Edition)., 2020.58(3):627–633 LIJian JIANGNan BAOYINBate Spatial Color Clustering Algorithm and Its Application in Image Feature Extraction Journal of Jilin University(Science Edition) 2020 58 3 627 633 Search in Google Scholar

Hou Yuanyuan, He Ruhan, Liu Junping, FUSION COLOR FEATURE AND DEPTH FEATURE CLOTHING IMAGE RETRIEVAL ALGORITHM. Computer Applications and Software., 2020.37(10):194–198 HouYuanyuan HeRuhan LiuJunping FUSION COLOR FEATURE AND DEPTH FEATURE CLOTHING IMAGE RETRIEVAL ALGORITHM Computer Applications and Software 2020 37 10 194 198 Search in Google Scholar

Articoli consigliati da Trend MD

Pianifica la tua conferenza remota con Sciendo