Image processing approach for grading tobacco leaf based on color and quality
Article Category: research-article
Pubblicato online: 16 dic 2019
Pagine: 1 - 10
Ricevuto: 30 gen 2019
DOI: https://doi.org/10.21307/ijssis-2019-010
Parole chiave
© 2019 Agus Harjoko et al., published by Sciendo
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Tobacco (
In Indonesia, tobacco cultivation occupies 0.37% of total agricultural land. The production reaches 196,154 tons of tobacco in 2016. That what makes Indonesia ranked fifth in the world (FAO, 2018b). However, the gap in the production of tobacco is too far compared to China with 2,805,615 tons or India with 761,318 tons. One of the reasons is the tobacco leaf grading system in Indonesia still done manually, relies on the skill and experience of tobacco leaf graders. This case is found in PT. Perkebunan Nusantara X (PTPN X, 2016), one of the tobacco manufacturer company in Indonesia. PTPN X plants Virginia type of tobacco leaf to target the cigarettes export market. The grading system classifies tobacco leaf into three categories with eight classes per category and done manually by the workers. The decision made by the workers is subjective, depend on their skill and experience, which often leads to false classification. Moreover, the lighting condition on its grading chamber relies on sunlight that susceptible to illumination changes.
Large tobacco plantation needs many tobacco leaf graders. In order to become a skilled grader, the worker needs to be trained. The training will take a long time and substantial cost to produce sufficient graders. Even if the plantation has enough graders, the monotonous and long duration of work will raise the human error.
The high demand for tobacco leaf triggers the mechanization of tobacco manufacture to push mass production. Therefore, many types of research that support tobacco manufacture especially the tobacco leaf grading system are conducted and sponsored by agribusiness companies. Most of the research works employ image processing techniques to determine the quality of tobacco leaf (Ni et al., 2009; Zhang and Zhang, 2011; Liu et al., 2012; Mallikarjuna and Guru, 2013; Bin et al., 2016; Jianqiang et al., 2018). It was expected because image processing techniques have been widely used in agriculture to processing post-harvest agricultural products (Chen et al., 2002; Laykin et al., 2002; Mashithoh, 2013).
In the study of Zhang and Zhang (2011), image processing techniques were used to extract and analyze color, size, shape, and surface texture on a flue-cured tobacco leaf. The tobacco leaf grading system used fuzzy comprehensive evaluation and resulted in 72% classification accuracy. In the study of Mallikarjuna and Guru (2013), a feature-level fusion was proposed for the classification of tobacco leaf. They used Haar wavelets, and gray-level local texture patterns (GLTP) that fused using concatenation rule then applied sequential backward selection (SBS) to select the discriminative texture features. K-nearest neighbor was used to classifying the tobacco leaves into viz, ripe, unripe, and over-ripe.
Liu et al. (2012) used image processing and generalized regression neural networks for tobacco leaf grading. They utilized digital image technology including near-infrared spectrum to extract area, perimeter, length, width, colors as a set of features that determined tobacco leaf quality. The features were classified using a generalized regression neural network into several grades. Near-infrared (NIR) spectroscopy for tobacco leaf classification also used in the studies of Ni et al. (2009), Bin et al. (2016), Jianqiang et al. (2018). NIR spectroscopy was rapid, non-destructive, and straightforward for tobacco leaf. NIR spectra were used to predict the content of some chemical components which are essential indicators to determine the tobacco leaf quality.
This research employs image processing techniques such as image thresholding, morphological operation, blob detection, and color analysis for grading tobacco leaf. This work focuses on grading tobacco leaf based on color and quality. The method covers the quality inspection of tobacco leaf, namely, leaf defect detection and classification of tobacco leaf based on color. The rest of this paper is organized as follow: Section “Tobacco leaf grading system” presents the proposed tobacco leaf grading method, Section “Image processing approach for grading tobacco leaf” presents the results and discussion, and Section “Result and discussion” presents the conclusion of this work.
Indonesia has a standardized grading system for tobacco leaves. Tobacco manufacture company such as PTPN X trains their graders according to the manual (PTPN X). The standard procedure of grading tobacco leaf consists of: grading based on leaf position; grading based on color and quality; grading based on usage; and grading based on a detailed color level.
Figure 1 shows the illustration of tobacco cultivation. The lower tobacco leaf can be harvested when the flowering started, and the upper leaf can be harvested afterward.
Figure 1.
Illustration of tobacco cultivation (FAO, 2018a).
The leaf position is used to grade tobacco leaf as shown in Figure 2, namely (A) KOS, (B) KAK, and (C) TNG. KOS category consists of a lower tobacco leaf that grows near the ground field. The leaf is broad with the rounded tip, thin, and has smooth leaf veins. KAK category consists of middle tobacco leaf. The leaf is full of a slightly narrowed tip, slightly thick, and has smooth leaf veins. TNG category consists of upper tobacco leaf. The leaf is wide with narrowed tip, thick, and has slightly broken leaf veins.
Figure 2.
Tobacco leaf categories based on leaf position.
After tobacco leaf is grouped based on leaf position, abnormality detection is applied to find diseases, stripes, and spots on the leaf surface and defect detection are applied to find holes or torn trace on the leaf surface. The leaf defects are break down into four categories based on the holes or torn position as shown in Figure 3, namely (A) top, (B) bottom, (C) side, and (D) center. This leaf defects category is one of the criteria to grade tobacco leaf based on color and quality.
Figure 3.
Leaf defect categories based on holes or torn positions.
The next step is to classify tobacco leaf based on maturity level and color. The maturity level of tobacco leaf is categorized as MT for young/unripe leaf, T for mature/ripe leaf, and TT for the old/over-ripe leaf. In this research, we focus on the mature/ripe category. The mature/ripe category is broken down into three sub-categories based on leaf color as shown in Figure 4, namely (A) K (yellow), (B) M (brown), and (C) B (green). This category is one of the criteria to grade tobacco leaf based on color and quality.
Figure 4.
Mature/ripe leaf subcategories based on the color of tobacco leaf.
In this research, image processing techniques are applied for grading tobacco leaf based on color and quality. The proposed method consists of tobacco leaf image acquisition, leaf area segmentation, leaf defects detection, and color-based tobacco leaf classification. The procedure is explained in the following steps: Tobacco leaf image acquisition Tobacco leaf is captured using a camera in a container with a fixed resolution, standardized lighting, and uniform colored leaf bed. The camera is configured manually to turn off automatic settings such as autofocus, autoexposure, and auto white balance. The lighting is positioned vertically above the leaf bed and configured to produce constant light and to spread the light evenly. The leaf bed color is dark gray and made using material that minimizes the light’s reflectance. Leaf area segmentation Leaf area segmentation uses image thresholding in hue channel, blob detection, and morphological operation. The procedure to segment the leaf area is shown in Figure 5 and explained in the following steps: The tobacco leaf image is converted to HSV color space. The tobacco leaf area is segmented using image thresholding in the hue channel with a given threshold. The threshold is obtained from the experiment. Blob detection is performed on the binary image of the leaf area to extract blob’s contour, position, and orientation. The blob detection algorithm is set to retrieve only external contours and reject the inner contours as we were only interested in the largest blob area. The blob with the largest area is designated as potential leaf area. Leaf shape is rotated according to the tilt degree of leaf major axis to fix the leaf orientation. The morphological operation is applied to fill the holes or torn on the leaf surface. The leaf image is cropped to fit the leaf area and remove the background. Leaf geometry such as size and dimension is calculated. Finally, the result of leaf area segmentation is a leaf area image with normalized leaf orientation and leaf geometry information. Leaf defects detection The defect on the leaf surface reduces the quality of tobacco leaf. Leaf defects are categorized based on the defect location on the leaf surface, namely top, side, center, and bottom category as shown in Figure 6. From Figure 6, the center area has ellipse shape which proportional to the leaf size, the top part is on the leaf tip, the bottom part is on the leaf base, and the rest belong to side category. The procedure to detect leaf defects using image processing techniques is explained as follow: The morphological operation is applied to fill the leaf area. The difference between the filled leaf area and the original leaf area is computed to find holes. The holes are the potential of leaf defects. Blob detection is performed to find contour, area, center, and position of the defects. We simplify the blob detection algorithm to retrieve only external contours and reject the inner contours. Ignore all blobs that have an area smaller than a given threshold since it will not affect the quality of the tobacco leaf. In this research we only measure the quality of tobacco leaf from the size and position of leaf defects that represent by the appearance of holes on the leaf surface. The holes are detected from the difference between the leaf bed and leaf surface. Therefore, it will reduce the false detection of holes that detected as spot or disease. For every defect found: Check if the defect center point is inside the area of the center category using (1) where ( Otherwise, check if the defect center point is in the top area or the bottom area. If true, the defect belongs to the top or bottom category. Otherwise, the defect belongs to the side category. The size of the defect is calculated to know the level of damage on the leaf surface. Color-based tobacco leaves classification In the mature/ripe category, tobacco leaf is classified into three categories based on the color of the tobacco leaf. The category is determined using a set of thresholds in the hue and value channel. The procedure to classify tobacco leaf into K, M and B category is explained as follows: The tobacco leaf image is converted to HSV color space. The image is divided into small patches with fixed size as illustrated in Figure 7. We divide the tobacco leaf into small patches to handle color diversity that will likely affect classification using a global threshold. The weight of each patch is calculated by dividing the leaf area inside the patch with the total number of pixels in the patch. The average pixel of hue and value channels in each patch is calculated. A set of the threshold is applied to classified each patch into a local K, M, and B category based on average hue and value. The illustration is shown in Figure 8 where The leaf color category is determined from the highest vote of color categories. The vote is calculated from the sum of the weight of each patch. Tobacco leaf on each color category is divided into eight classes from 1 to 8 (bright to dark). The classes are classified using a set of threshold, namely
Figure 5.
The procedure of leaf area segmentation.
Figure 6.
Area of leaf defects category.
Figure 7.
Small patches applied on the leaf area.
Figure 8.
A set of the threshold to determine the color category.
Figure 9.
A set of the threshold to determine the class in each color category.
The proposed tobacco leaf grading method is simulated in Matlab and runs on Intel Core i-7 6700K processor, 16 GB of RAM, and NVidia GTX 1050. The result, discussion, and parameters used in the experiment are explained as follows.
In order to grade tobacco leaf based on color and quality, tobacco leaf needs to be segmented from the background. Image thresholding, blob detection, and morphological operation are employed to segment the leaf area. From observation, the range of hue pixel in the tobacco leaf image is 15 ≤ hue pixel ≤ 80 in the range of 0 to 180 degrees (normally is 0-360 degrees but it is converted to fit 1-byte size for each pixel). Figure 10 shows the hue value that used as a threshold to segment leaf area.
Figure 10.
The threshold of hue pixel to segment leaf area.
The result of leaf area segmentation is shown in Figure 11 where (A) is the input tobacco leaf image, (B) is the result of image thresholding on hue channel, (C) is the result of normalized leaf orientation, and (D) is the result of final leaf area. From Figure 11B, the ellipse approximation of the tobacco leaf area shows a tilted orientation. By rotating the leaf according to the degree of tilted on leaf major axis, the orientation is normalized (see Figure 11C). The leaf orientation needs to be normalized to provide an easier estimation of defect position and leaf division into 10 × 6 small patches.
Figure 11.
The result of a tobacco leaf area segmentation.
After tobacco leaf area segmentation, leaf defects detection is performed to determine the defect location on the leaf surface. The method uses morphological operation, blob detection, and blob analysis. Based on the leaf defects category, two parameters, namely, defect location and level of damage are used to determine the tobacco leaf quality. The level of damage is calculated from the size of the defect area divided by the whole leaf area. The larger level of damage means less quality of tobacco leaf. Figure 12 shows the result of leaf defects detection where (A) is the tobacco leaf image, (B) is the binary image of the original leaf area, (C) is the binary image of filled leaf area, and (D) is the result of leaf defects detection. From Figure 12D, a hole is located inside the area of center category (cyan ellipse marker) with 0.29432% level of damage. The method to detect defect on tobacco leaf is tested on several samples of defected tobacco leaf and shows 0.0 of the log average miss rate and 1.0 of average precision rate which means our proposed method identifies the defect on tobacco leaf correctly.
Figure 12.
The result of leaf defects detection.
In order to classify tobacco leaf based on color, the tobacco leaf image is divided into small patches. Small patches are used to handle the non-uniform color of tobacco leaf produces by abnormal leaf surfaces such as diseases, stripes, or spots. Either large or small patches have their own contribution or vote (weight) to the classification. Since weight is the ratio between leaf surface area that covered by the patch and the size of the patch, the choice of patch size will not significantly affect the final classification. We perform some experiments with variation of patch sizes such as 8 × 4, 10 × 6, and 12 × 8. The result shows that 10 × 6 patch size achieved the highest classification accuracy with small margin compare to the other patch size. Small patch size means larger area inside the patch while large patch size means a smaller area inside the patch. It will affect the strength of local features used for classification.
The average pixel of the hue and value channel in each patch is calculated to be classified into local K, M, B category using a set of threshold (see Figure 8). Table 1 shows the threshold derived from analysis of hue-value features distribution of SL (Special Lot)-type tobacco leaf training data. Figure 13 shows the scatter plot of the distribution of the hue-value features where red dots are the distribution of M category, yellow dots are the distribution of K category, and green dots are the distribution of B category. The features distribution of each category is mostly clustered, make it easier to classify using simple threshold. We also consider using experimental threshold rather than automatic threshold because from the observation in tobacco manufacture company, a different type of tobacco leaf requires a different set of the threshold. According to PTPN X, the color of tobacco leaves that belong to the same type can be vary, depends on the harvest time or weather condition. Moreover, the set of threshold can be obtained using a small number of training data from the boundary of each category and more comfortable to configure and standardized by the operator.
The threshold for local color-based classification.
| Threshold | Hue | Value |
|---|---|---|
| Lower | 15 | 60 |
| Middle | 34 | 127 |
| Upper | 80 | 200 |
Figure 13.
The distribution of hue-value features of a color-based tobacco leaf category.
The local color category is calculated on each small patches using the set of thresholds in Table 1. We use the voting system to determine the global color category from the local color category of each patch. Each patch contributes to the voting system based on its weight. Figure 14 shows the result of tobacco leaf image classification by majority voting (A) tobacco leaf is classified to M category with 34.3444 votes of M category from total 37.1511, (B) tobacco leaf is classified to K category with 28.775 votes of K category from total 37.4557, and (C) tobacco leaf is classified to B category with 28.9666 votes of B category from total 35.8633.
Figure 14.
The result of color-based tobacco leaf classification.
The proposed method tested on 204 tobacco leaves of SL (Special Lot) type. The classification result is shown in Table 2 using a confusion matrix. The accuracy, precision, sensitivity, and specificity score are calculated to measure the performance. The proposed method achieves 91.667% accuracy, 91.667% precision, 89.519% sensitivity, and 95.751% specificity score. False-positive occurs on the border of every category or near the threshold, especially in K and B category. From Figure 8, the only difference between K and B category is the value pixel, and it is sensitive to illumination. However, due to the standardized lighting environment that used in the tobacco leaf image acquisition process and by using a set of threshold that shown in Figure 8, the proposed method can easily handle color gradation to grade tobacco leaf based on detailed color level.
Performance evaluation of color-based tobacco leaf classification.
| Actual | ||||
|---|---|---|---|---|
| M | K | B | ||
| M | 33 | 1 | 2 | |
| Predicted | K | 8 | 72 | 4 |
| B | 2 | 0 | 82 | |
| Accuracy | 0.91667 | |||
| Precision | 0.91667 | |||
| Sensitivity | 0.89519 | |||
| Specificity | 0.95751 |
This research proposes a tobacco leaf grading method based on color and quality. The method employs image processing techniques such as image thresholding, morphological operation, blob detection, blob analysis, and color analysis. The result shows that the proposed system can accurately classify tobacco leaf with 91.667% accuracy into K (yellow), M (brown), and B (green) color categories. The classification method uses a set of the threshold to determine the local color category in small patches of the tobacco leaf image. The global color category is determined using majority voting from each patch. For future work, the method will cover more procedures from tobacco leaf grading system such as grading tobacco leaf based on leaf position and grading tobacco leaf based on a detailed color level.