Journal of Food Engineering 88 (2008) 364372

Contents lists available at ScienceDirect

Journal of Food Engineering

journal homepage: www.elsevier.com/locate/jfoodeng

Robust color space conversion and color distribution analysis techniques for date maturity evaluation
Dah-Jye Lee *, James K. Archibald, Yu-Chou Chang, Christopher R. Greco
Department of Electrical and Computer Engineering, Brigham Young University, 459 CB, Provo, UT 84602, United States

a r t i c l e

i n f o

a b s t r a c t
Color is important in evaluating quality and maturity level of many agricultural products. Color grading is an essential step in the processing and inventory control of fruits and vegetables that directly affects profitability. Dates are harvested at different levels of maturity that require different processing before the dates can be packed. Maturity evaluation is crucial to processing control, but conventional methods are slow and labor-intensive. Because date maturity level correlates strongly with color, automated color grading could be used. A novel and robust color space conversion and color index distribution analysis technique for automated date maturity evaluation that is well suited for commercial production is presented in this paper. In contrast with more complex color grading techniques, the proposed method makes it easy for a human operator to specify and adjust color preference settings for different color groups representing distinct maturity levels. The performance of this robust color grading technique is demonstrated using date samples collected from eld testing. 2008 Elsevier Ltd. All rights reserved.

Article history: Received 1 November 2007 Received in revised form 1 January 2008 Accepted 19 February 2008 Available online 4 March 2008 Keywords: Machine vision Color space conversion Color grading Date maturity evaluation

1. Introduction A typical date packing facility in the United States packs between 45 and 175 million individual dates each year. Date growers harvest all fruit from their orchards at the same time, yielding dates with varying levels of maturity. After harvesting, the fruits currently undergo manual inspection to determine if they are ripe and ready for packaging or if further processing is required. Unripe dates are sent to gain moisture in a hydrating building (most mature), to ripen in the sun (medium), or to dry in a heated building (least mature). After drying, most dates ripen and become ready for packaging. However, insufcient drying can cause dates to rot and turn sour, while excessive drying can cause the fruit skin to peel, lowering the fruit quality and decreasing its value. It is therefore critical that the maturity of each harvested date be evaluated accurately so that it can be processed appropriately. Depending on the maturity level, date colors range from yellow to dark red with many shades in between. The strong correlation between maturity level and color allows experienced sorters to determine maturity level by visually examining the color of the fruit, a process used by Bard Valley Medjool Date Grower Association in California, USA. This important but labor-intensive sorting process constitutes a major expense for packers and growers. A ro-

* Corresponding author. Tel.: +1 801 422 5923; fax: +1 801 422 0201. E-mail address: djlee@ee.byu.edu (D.-J. Lee). 0260-8774/$ - see front matter 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.jfoodeng.2008.02.023

bust color grading technique could automate sorting and reduce costs. Machine vision technology has been employed in many agricultural applications involving color grading. Machine vision systems for real-time color classication (Lee and Anbalagan, 1995; Lee, 2000; Zhang et al., 1998) have been commercialized to grade food products based on color. Other agricultural applications include the color grading of fresh market peaches (Nimesh et al., 1993; Miller and Delwiche, 1989a,b; Singh et al., 1992), apples (Varghese et al., 1991; Hung et al., 1993), potatoes (Tao et al., 1995), peppers (Shearer and Payne, 1990), cucumbers (Lin et al., 1993), tomatoes (Choi et al., 1995), and dates (Janobi, 1998). Many of these systems have shown very promising results. Color sorting machines for apples, potatoes, and tomatoes are commercially available. To improve consistency, most color-grading systems convert color images captured in red, green, and blue (RGB) color components to a hue-saturation-intensity (HSI) representation. Processing decisions are then made primarily on the basis of color hue. Typically, the hue of the product is compared against reference values to determine its color grade (Miller and Delwiche, 1989a,b). Unfortunately, neither RGB nor HSI color spaces are particularly well suited to color grading. Of greatest concern is that selected reference colors seldom represent a convenient or intuitive partitioning of the multi-dimensional color space. In particular, it is difcult to dene and adjust the boundaries between adjacent color grades and to analyze color distribution. This problem is discussed in detail in Section 2.1.

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

365

An effective, user-friendly method to categorize the maturity levels of dates based on an analysis of their colors is presented in this paper. This method does not require complicated machine learning or articial intelligence algorithms. It allows the user to obtain results in a manner similar to human color perception by making simple adjustments to color boundary thresholds. For example, the method allows the threshold for a particular grade to be adjusted so that it includes fruit that is slightly darker red or brighter yellow. In combination with the proposed color distribution analysis capable of handling partial colors, this robust color space conversion method has been applied in commercial production to evaluate date maturity levels with very high accuracy. 2. Materials and methods 2.1. Problem and motivation To maximize the value of each unripe date, it is vital that it undergo an appropriate period of drying or hydration. The challenge is to separate dates into different maturity levels with a high degree of accuracy. Fig. 1 shows 12 date samples classied according to the seven maturity levels used by the Bard Valley Medjool Date Grower Association. These include: Ready (Fig. 1a and b), Skin (Fig. 1c), Light Red or Orange (Fig. 1d and e), Partial Red (Fig. 1f), Yellow Head/Yellow Spot (Fig. 1g and h), Partial Yellow (Fig. 1i and j), and Yellow (Fig. 1k). With the exception of an eighth grade, Trash (Fig. 1l), consisting of dried-out dates easily distinguished by their size, all maturity levels can be identied using color. In general, the less mature the fruit, the more yellow its color, but accurate grading is challenging. The rst two categories, Ready and Skin, are at the same maturity level and both can be packed immediately for sale; mature fruit with visible skin separation is placed in the Skin category.

Fruit classied as Light Red or Orange is usually soft, wet, and not fully mature; dates in the Partial Red class are slightly less mature. Fruit in the Yellow Head/Yellow Spot category is mostly orange or red but with small yellow spots. Fruit classied as Partial Yellow is mostly yellow but with small areas of orange or red. Dates in the Yellow category are the least mature and almost entirely yellow in color. 2.2. Challenges and requirements Typical computer vision algorithms such as image segmentation and object recognition work in a color space that interprets and models colors in a unique way. Several color spaces exist, but none is superior for all applications (Hunt, 1995; Fairchild, 1998). For color grading, RGB and HSI are the most commonly used. RGB is the most studied color space because electronic images are often acquired in terms of RGB components. Because it is based on human-distinguishable hues, the HSI color space is a more intuitive representation than RGB. Color grading in the RGB color space is complicated by the 3D representation of colors, making the selection and adjustment of color preferences difcult. Fig. 2 illustrates this difculty with a plot of color values from the 12 sample images in Fig. 1. As can be seen, the dominant color associated with each separate grade corresponds to a spatially distinct 3D volume. Because adjacent grades correspond to regions that are not contiguous, it is difcult for an operator to make even slight adjustments to classication thresholds during production, as values on all three axes must be modied simultaneously. Moreover, the 3D nature of the RGB space makes it difcult to perform the analysis of fruit color distributions required by applications such as date maturity evaluation. Most color grading systems use hue rather than RGB values to specify color preferences, but HSI has its own disadvantages. First,

Fig. 1. Dates at different maturity levels. Ready for packing (a) and (b); Slight skin separation (c); Soft with Light Red or Orange color (d) and (e); Partial Red (f); Yellow Head (g); Yellow Spot (h); Partial Yellow (i) and (j); Yellow (k); Trash (l). (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

366

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

yellow) and analysis of color distribution are more intuitive and much simpler than in multi-dimensional color spaces. 2.3. Novel color space conversion algorithm Most color cameras used for machine vision and automation output RGB signals. Individual RGB channels are usually digitized into 8-bit values (0255), so 24 bits are required to represent the color value of each pixel. Although 24 bits allow the specication of more than 16 million unique colors, the range of observed colors for any grading application will likely be much smaller. If the range of these colors of interest is well dened for a given application, color grading can be signicantly simplied. For date maturity level evaluation, as shown in Fig. 3a, colors range from yellow to dark red, with orange and light red in between. Many colors such as green and blue will not occur. Given a set of specic colors of interest for the product to be graded, it is possible to map the color range to a 1D color space such that the resulting index value reects increasing product preference. The focus of this paper is a novel and robust color space conversion technique that achieves such a mapping. The proposed patent-pending technique meets the two requirements for userfriendly color grading identied in the previous section, and it is efcient, effective, and easy to implement. The proposed new color space conversion maps limited colors of interest in 3D RGB color space into 8-bit values in the resulting linear color space. Only a small subset of the full range of 16M possible colors can be represented by the resulting 256 levels, but this is sufcient to represent the colors of interest for many applications. (If higher color resolution is required, 16-bit integers can be used.) The formula for this conversion is: Linear color value c1 RGB c2 R2 c3 G2 c4 B2 c5 RG c6 RB c7 GB c8 R c9 G c10 B c11 1

Fig. 2. Separating colors for date maturity evaluation in 3D RGB color space. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

the set of expected colors for a given application will not necessarily correspond to contiguous hue values. Hue is generally normalized to lie between 0 and 360, with red represented as 0, green as 120, and blue as 240. Nearly all formulations of HSI require segment-specic computations for each 60 segment of the hue circle, but this introduces visible discontinuities into the color space that make it difcult for a user to set or adjust color grade boundaries using hue values alone. Another disadvantage of using hue for color grading is that it does not contain brightness information a dark red with low intensity can have the same hue value as a much brighter red. This can be overcome by adding the intensity component from the HSI color space in the computations, but setting and adjusting color grade boundaries in the resulting 2D space (hue, intensity) encounters the same problem arising in the 3D RGB space: adjacent color grades are likely to correspond to noncontiguous regions and adjustments to grading thresholds (region boundaries) will not be intuitive. Moreover, analysis of color distribution in 2D would add unwanted complexity to the system. At the outset of this project, two major requirements for userfriendly color grading were identied: (1) easy determination of color preference settings and (2) easy adjustment of these settings to match the desired color separations. Based on the sample images in Fig. 1, Fig. 3a shows the color range of dates at all observed maturity levels. Ideally, an operator could modify the color preferences simply by adjusting the single threshold on this scale that separates two adjacent grades. This can be achieved by converting each color in 3D RGB space into a single numerical value corresponding to its position or index within the color range. Fig. 3b shows the 1D color indices (represented in grayscale values) corresponding to the colors in Fig. 3a. Dark red colors are converted to high index values; orange and yellow colors result in low values. With this new color space conversion, adjustments to color grade boundaries (e.g., making a particular category more red or more

where R, G, and B are the 8-bit color channel values of a given pixel in the original image, and each ci, 1 6 i 6 11, is an application-specic weight whose derivation will be explained shortly. As can be seen, a third-order term (R G B) and the full rank of second-order polynomials are employed to convert the original 3D information to a 1D color index for each pixel. The 11 weights or coefcients in the formula are obtained through calibration using RGB values of a selected set of colors of interest and a preferred linear color index for each. To solve for the 11 coefcients, at least 11 sample colors must be selected from the applications color range, each with distinct RGB values. Eq. (1) with multiple sample colors can be written in matrix form as shown in the following equation: 2 R1 G1 B1 R2 1 R2 2 . . . G2 1 G2 2 . . . B2 1 B2 2 . . . B2 n R1 G1 R2 G2 . . . Rn Gn R1 B1 R2 B2 . . . Rn Bn G1 B1 G2 B2 . . . Gn Bn R1 R2 . . . Rn G1 G2 . . . Gn B1 B2 . . . Bn 1 3

Rn Gn Bn R2 G2 n n 3 2 3 v1 C1 6C 7 6v 7 6 27 6 27 6 . 76 . 7 6 . 7 6 . 7 4 . 5 4 . 5 2 Cn vn

6 6 R2 G2 B2 6 6 . 6 . 4 .

7 17 7 7 . 7 . .5 1 2

Fig. 3. (a) Date colors at different maturity levels and (b) their converted color indices. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

In the equation, Ri, Gi, and Bi (i = 1, . . ., n) are the RGB values for a given color sample and vi is the corresponding desired linear color index. Given RGB and v values, the coefcients can be determined by solving (2) using a least-squared error method. Singular value decomposition (SVD) (Moon and Stirling, 2000) is a convenient solution methodology because a single solution of Eq. (2) may not

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

367

exist due to imperfect selection of sample colors and desired color indices. Moreover, SVD allows the use of more than 11 sample colors in solving for the optimal solution. Once the coefcients are known, all image pixels can be mapped to the linear color indices used for grading. Fig. 4 illustrates the essential idea of the proposed new color space conversion technique. Fig. 4a shows the range of colors of interest for date maturity levels in 3D RGB color space, and Fig. 4b shows (in 256-level grayscale) the corresponding index values in the new 1D color space. Note that lower values represent yellow fruit, while higher values represent more mature fruit. The color tube in Fig. 4a would effectively be widened if more colors were observed in dates. In converting to the linear space using Eq. (1), observed colors that lie outside the tube of pre-selected colors will be assigned an index value corresponding to the closest RGB value within the tube. 2.4. Image processing Fruit must be separated or segmented from the background of the image before its color can be analyzed. Blue plastic material was chosen to build the fruit carrier for imaging for two reasons. First, blue does not occur naturally in dates. Second, blue is one of the three channels in the RGB color space, making it easier to lter than colors that require multiple channels to represent. Fig. 5 shows an example image with two lanes of the fruit carrier. Only the area within each green box is analyzed in grading the fruit. The fruit is segmented from the blue background by creating masks corresponding to the fruit area using the blue and red channels. The proposed color space conversion method is then used to remove areas of shadow and increase the accuracy of the mask.

Fig. 6 summarizes the segmentation steps, and Fig. 7 shows results on sample images. Fig. 7a shows the original RGB image. The blue and red channels (Fig. 7b and c, respectively) are rst binarized to generate blue and red masks. For each pixel in the image, the corresponding value in the blue mask (Fig. 7d) is set to 0 if the value of the blue channel is high (blue background), otherwise it is set to 255 (fruit). Values in the red mask (not shown) are set to 255 if the red channel is high (fruit), otherwise they are set to 0. These two masks are then combined using a logical binary OR operation to create an intermediate mask (Fig. 7e) without the holes and missing areas commonly seen in the individual blue and red masks (see Fig. 7d). By using the logical binary AND operator on corresponding pixel and mask values, the intermediate mask is used to select the fruit area from the original RGB image (Fig. 7f).

Fig. 5. Date image with a bright blue background. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

Fig. 4. 3D representation of (a) colors of interest for date maturity evaluation and (b) converted continuous color indices. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

Fig. 6. Fruit are segmentation algorithm owchart.

368

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

2.5. Color index distribution analysis As shown in Section 2.1 and Fig. 3, the colors of interest for this application are limited. However, as Fig. 1 shows, different maturity levels exhibit distinct color patterns or distributions, some of which involve mixed colors. A simple evaluation method using average color or color variation can be effective in other grading applications, but it is not sufcient for date maturity evaluation. A color index distribution analysis method is developed to obtain better separation of different maturity levels. This method is performed on color index values produced using the proposed color space conversion technique. Because this method analyzes colors within a 1D color space, it is much simpler than analysis in a 2D or 3D space, as discussed in Section 2.2. Fig. 8 shows the color index distributions of the rst 11 sample images shown in Fig. 1 (excluding the Trash grade which is detected by fruit size). Plotted in two separate gures for clarity, these histograms were obtained from the converted color indices using the segmentation and color space conversion algorithms described above, in the same manner that Fig. 7j was obtained from the original image. The histograms can be roughly divided into four groups: 1. Ready and Skin: This group has high color indices (red colors). The Skin category has more variation (wider distribution) than Ready. 2. Orange and Partial Red: This group has two histogram peaks. The two categories can be distinguished from each other using the distance between the two peaks and color index variation. 3. Yellow Head and Yellow Spot: This group also has two peaks but with color index values much lower than the second group. Yellow Head and Yellow Spot are considered to have the same maturity level. 4. Partial Yellow and Yellow: This group may have one or two peaks. The rst results from a large number of yellow pixels; the second exists only if the fruit is partially yellow, and it represents the portion of the date that is red or orange. The presence of the second peak can distinguish between the two categories. The proposed color index distribution analysis consists of three steps: histogram smoothing, peak detection, and calculations of color value and consistency. A 5 1 average lter is used to smooth the histogram to avoid false detection of peaks. A 1D 11 1 Laplacian operator is then applied to detect each zero-crossing of the histogram as a peak. The Laplacian operator is implemented as shown in the following equation:
jX i1 ji5 jX i5 ji1

Fig. 7. Image segmentation results. (a) Original image, (b) blue channel, (c) red channel, (d) binary image of blue channel, (e) binary image of blue ORed with binary image of red, (f) segmented fruit image using (e) as a mask, (g) color indices of (f), (h) binary image after ltering out dark blue background, (i) segmented fruit image using (h) as a mask, and (j) color indices of (i). (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

At this point, pixels in the fruit area can be converted to color index values using the proposed color space conversion method (Fig. 7g), but it can be seen (e.g., lower left of top date, upper left of bottom date) that the fruit area includes pixels from shadows cast by the fruit on the carrier that are close in color to dark red fruit. It is desirable to remove these shadow pixels before color analysis as they would adversely affect grading accuracy. Using a technique described in Section 3.1, the system is calibrated in such a way that these shadow colors are converted to a color index of 255 (visible in Fig. 7g). As the owchart in Fig. 6 shows, a nal mask (Fig. 7h) is generated by removing areas of shadow from the intermediate mask. This is accomplished by setting mask pixels to 0 if the corresponding pixels in Fig. 7g have a color index of 255 (shadow). Occasionally, some dark red pixels inside the fruit area are removed mistakenly because they are indistinguishable from shadow pixels, but these voids are lled in by a later step that includes connected component analysis. Using the nal mask with shadow areas removed, the fruit area can be accurately segmented (Fig. 7i) and the proposed color space conversion method can be used to obtain nal color indices, as shown in Fig. 7j.

zi hi 10

hj

hj

where h(i) and z(i) are the histogram and zero-crossing values at color index i, respectively. Color indices with zero-crossing values higher than a user-specied threshold are considered to be locations of histogram peaks. Only the highest two zero-crossings or peaks are considered. The last step of the novel color index distribution analysis involves calculations of color values and consistency. First, the standard deviation (StDev) of the converted color indices is computed. The color value is calculated as a weighted average of the histogram values within the color index range of peak StDev as shown in the following equation: , ipeak StDev ipeak StDev X X Color value hi i hi 4
ipeakStDev ipeakStDev

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

369

If the histogram has two peaks, then the color value of the fruit is calculated using Eq. (4) using the peak associated with the most frequently occurring colors. To illustrate, consider the two peaks of the Partial Red histogram in Fig. 8a, located at color indices 112 and 202. The midpoint of the two peaks is calculated (157), and the number of pixels below the midpoint is compared with the number above the midpoint. In this case, more pixels are below the midpoint, so the peak at 112 is used to compute color value using Eq. (4). Similarly, in the histogram of Yellow Head in Fig. 8b, peaks are identied at index values of 56 and 147 the former corresponds to the yellow spot and the latter to the most common color on the remainder of the fruits surface. Comparing pixel frequency above and below the midpoint (102) leads to the use of the peak at index 147. If the histogram has a single peak, then color consistency is calculated as 100-StDev, resulting in high consistency values when StDev is low. If the histogram has two peaks, the consistency measure must be reduced by more than the corresponding increase in the value of StDev. This makes it possible to distinguish between, say, a fruit that is fully mature and a fruit that is entirely mature

except for one yellow spot. The adjusted color consistency is calculated as follows: Color consistency 100:0 StDev 1:0 jpeak1 peak2j=255 5

where peak1 and peak2 are the index values of the two detected peaks. Note that Eq. (5) works for histograms with a single peak by setting peak2 equal to peak1. 3. Experimental results and discussion 3.1. Calibration results Thirteen sample colors ranging from red to yellow were used to calibrate the system for date maturity evaluation. The colors are displayed in Fig. 9a, and their RGB values are plotted in Fig. 9b. This set of 13 RGB values was selected to cover the full range of colors at all maturity levels. The values were used to create a 13 11 matrix as shown in Eq. (2), with one row per sample color. The colors

a
200 Ready 1(a)

Color Index Histogram

Ready 1(b) Skin 1(c) Orange 1(d) Orange 1(e) Partial Red 1(f)

150

Pixel Count

100

50

0 1 21 41 61 81 101 121 141 161 181 201 221 241

Color Index

b
Yellow Head 1(g) 250

Color Index Histogram

Yellow Spot1(h) Partial Yellow 1(i) Partial Yellow 1(j) Yellow 1(k)

200

Pixel Count

150

100

50

0 1 21 41 61 81 101 121 141 161 181 201 221 241

Color Index
Fig. 8. Histograms of the converted 1D color index values of the sample images shown in Fig. 1: (a) for Fig. 1af and (b) for Fig. 1gk. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

370

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

(rows) were arranged in the order of color preference, from least mature to most mature fruit. For each color sample, a human expert specied a desired linear color index in the range from 0 to 235. (Index values for fruit colors were limited to 235 to distinguish from shadow pixels on the blue carrier that are a darker red and will be assigned index value 255.) The resulting values were arranged

b
250

RGB Values of Sample Colors

Red Green Blue

200

into a 1 13 matrix with values 18 36 54 72 90 108 126 144 162 180 196 220 235T corresponding to vector v in Eq. (2). The 13 equations were then solved using SVD, resulting in the 11 coefcient values: [0.000955, 0.000541, 0.018773, 0.25405, 0.025364, 0.055644, 0.391265, 0.510995, 10.321283, 2.867669, 0.080986]. These coefcients were used to convert the 13 sample colors to color indices to evaluate the conversion accuracy. Fig. 10a depicts the resulting index values on a 256-level grayscale. As can be seen, the samples transition smoothly from black to white. Fig. 10b shows the resulting indices (in red) and the desired indices (in blue) for the 13 samples. Discrepancies between the two values result from conversion inaccuracies in the mapping process. For the 13 data points, the average error was 2.82 units on the color index scale. This error rate is sufciently low to perform very accurate color grading. The result demonstrates that the proposed approach using the third-order polynomial works very well for this application. In short, it converted a range of 24-bit colors of interest into linear indices that closely reect the color preferences of an experienced user.

150

100

50

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Fig. 9. (a) Sample colors used for calibration and (b) their RGB values. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

b
250

Desired and Converted Color Indices

Desired Converetd

200

150

100

50

0 1 2 3 4 5 6 7 8 9 10 11 12 13

Fig. 10. Color indices of the sample colors used for calibration: (a) desired color indices and (b) converted color indices using calculated coefcients. Sample 2 has the smallest error of 0.085 and sample 12 has the largest error of 8.31. The average error is 2.82. All units are on the color index scale. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

Fig. 11. Binary mask, segmented fruit, and converted color indices of sample images shown in Fig. 1. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

371

Color Index Distribution and Separation

Ready Skin Light Red or Orange Partial Red Yellow Head/Spot Partial Yellow Yellow 90

80

70

Color Consistency

60

50

40

30

20 220 200 180 160 140 120 100 80 60 40 20 0

Color Index
Fig. 12. Color index and color consistency distribution of the 700 date images used for testing. (For interpretation of the references in color in this gure legend, the reader is referred to the web version of this article.)

3.2. Color mapping and data collection Seven hundred Medjool dates at differing maturity levels were randomly selected for testing. Images of these 700 samples were taken using a color camera mounted in an enclosed light chamber to ensure consistent lighting. Grading was performed by rst segmenting the fruit area from the blue background using the proposed segmentation algorithm (described above) and then converting the segmented image into an 8-bit image in the new linear color space. The color grade was evaluated using a combination of color value and consistency, both of which were calculated as discussed in Section 2.5. Fig. 11 shows the nal fruit mask, segmented fruit area, and converted color indices of each of the images shown in Fig. 1. (Both gures present the dates in the same order.) The proposed fruit segmentation algorithm worked very well for all maturity levels, successfully ltering out the blue background and shadows. Of those samples shown, only the rst two (Fig. 1a and b) suffer from noticeable over-segmentation, resulting in a slightly smaller fruit area for color analysis. This over-segmentation occurred because the mature fruit colors are close to the color of shadows on the blue background. Because the segmented fruit area is only slightly smaller than the actual fruit area, this over-segmentation has little if any effect on grading accuracy. Fig. 12 shows the distribution of color and consistency for the 700 selected dates using the proposed color index distribution analysis technique. To achieve the stated goals of user-friendly control of color grade boundaries, the measurement space was di-

vided into rectangular regions, each determined by cutoff values for colors and consistency, and each representing a color grade or maturity level. For this collection of fruit, the best possible divisions of color and consistency values are shown by dotted lines in Fig. 12. During normal operation, with varying distributions of maturity levels and coloring, the operator can easily adjust the region associated with each grade by altering the appropriate thresholds. The system represents the color and consistency cutoff values as a 2D matrix. Representative settings are shown in Table 1, with values derived from Fig. 12. In the table, Grades AG represent Ready, Skin, Light Red or Orange, Partial Red, Yellow Head/Yellow Spot, Partial Yellow, and Yellow, respectively, and H represents everything else. For example, fruit with a color index higher than

Table 1 Typical color grading settings derived from Fig. 12 170 68% 62% 59% 57% A B D D D 105 C C D D D 63 C C C C E 5 F F F G G 0 H H H H H

Color index thresholds are shown across the top and consistency thresholds are down the left side. Fruit is assigned one of the letter grades according to the two thresholds.

372

D.-J. Lee et al. / Journal of Food Engineering 88 (2008) 364372

Table 2 Columns 1 and 2: Maturity levels and their letter grades shown in Table 1 Maturity Ready Skin Light Red/Orange Partial Red Yellow Head/Spot Partial Yellow Yellow Total # Grade A B C D E F G # of samples 188 82 56 103 106 71 94 700 # classied 185 74 50 101 79 62 85 636 Accuracy (%) 98.40 90.24 89.29 98.06 74.53 87.32 90.43 90.86

Column 3: number of samples for each grade. Column 4: number of samples correctly classied. Column 5: classication accuracy of each grade and the overall average accuracy.

170 is classied as Grades A, B, or D depending on its consistency (greater than 68%, between 68% and 62%, or below 62%). Fruit with a color index between 105 and 170 and color consistency above 62%, or with a color index between 63 and 105 and color consistency above 57%, is classied as Grade C (Light Red or Orange). Other entries in the table can be similarly interpreted. Using the settings from Table 1, the 700 Medjool samples selected for testing were classied, resulting in the grading accuracy shown in Table 2. Because it has consistent color and a simple histogram with a single peak, the Ready category has the highest grading accuracy of 98.4%. Yellow Head/Yellow Spot and Partial Red both have dual peaks and similar histograms. Yellow Head/ Yellow Spot can be misclassied as Partial Red if the color of the Yellow Head portion is close to orange. The overall grading accuracy is 636 out of 700, approximately 90.9%. The proposed color space conversion and color index distribution analysis techniques have been implemented and proven successful in evaluating date maturity in commercial production. A prototype system was built and tested in a packing facility in Arizona, USA. The system meets the maturity evaluation standard set by the Bard Valley Medjool Date Grower Association, and it meets the industry requirement of grading at least 20 pieces of fruit per second using a single commercial personal computer with two cameras for four processing lanes. 4. Conclusion A new color space conversion method and color index distribution analysis technique specically for automated date maturity evaluation has been presented. The proposed approach uses a third-order polynomial to convert 3D RGB values into a simple 1D color space. Unlike other color grading techniques, this approach makes the selection and adjustment of color preferences easy and intuitive. Moreover, it allows a more complicated distribution analysis of fruit surface colors. The user can change color

and consistency cutoff points in a manner consistent with human color perception, simply sliding a cutoff point to include fruit that is slightly darker or lighter red. Moreover, changes in preferred color ranges can be completed without reference to precise color values. Furthermore, by converting 3D colors to a linear color space, color distribution analysis required for date maturity evaluation is much more straightforward. The implementation of this new color space conversion method and the results presented demonstrate the simplicity and accuracy of the proposed technique. To calibrate the system, an experienced grader species a set of colors of interest, each accompanied by a preferred index value on a linear scale. Provided that the selected color samples cover the complete range of expected colors, accurate color grading will result. This new technique can be applied to other color grading applications that require the setting and adjustment of color preferences.

References
Choi, K.H., Lee, G.H., Han, Y.J., Bunn, J.M., 1995. Tomato maturity evaluation using color image analysis. Transactions of the ASAE 38 (1), 171176. Fairchild, M.D., 1998. Color Appearance Models. Addison-Wesley, Reading, MA. Hung, Y.C., Morita, K., Shewfelt, R., Resurreccion, A., Prussia, S., 1993. Color evaluation of apples. American Society of Agricultural Engineers 93 (6541), 15. Hunt, R.W.G., 1995. The reproduction of colour in photography. Printing and Television, fth ed. Fountain Press, England. Janobi, A., 1998. Color line scan system for grading date fruits. In: ASAE Annual International Meeting, Orlando, FL, USA, 1216 July. Lee, D.J., Anbalagan, R., 1995. High-speed automated color sorting vision system. In: SPIE Optical Engineering Midwest 95, Chicago, IL 2622, pp. 573579. Lee, D.J., 2000. Color space conversion for linear color grading. In: Proceedings of SPIE Intelligent Robots and Computer Vision XIX, Boston, MA 4197, USA, pp. 358366. Lin, W.C., Hall, J.W., Klieber, A., 1993. Video imaging for quantifying cucumber fruit color. HortTechnology 3 (4), 436439. Miller, B.K., Delwiche, M.J., 1989a. Automatic grading of fresh peaches using color computer vision. Acta-Horticulturae 254, 161166. Miller, B.K., Delwiche, M.J., 1989b. A color vision system for peach grading. Transactions of the ASAE 32 (4), 4841490. Moon, T.K., Stirling, W.C., 2000. Mathematical Methods and Algorithms for Signal Processing. Prentice Hall, Upper Saddle River, NJ. Nimesh, S., Delwiche, M.J., Johnson, R.S., 1993. Image analysis methods for real-time color grading of stonefruit. Computers-and-Electronics-in-Agriculture 9 (1), 71 84. Shearer, S.A., Payne, F.A., 1990. Color and defect sorting of bell peppers using machine vision. Transactions of the ASAE 33 (6), 20452050. Singh, N., Delwiche, M.J., Johnson, R.S., Thompson, J., 1992. Peach maturity grading with color computer vision. American Society of Agricultural Engineers 92 (3029), 23. Tao, Y., Heinemann, P.H., Varghese, Z., Morrow, C.T., Sommer, H.J., 1995. Machine vision for color inspection of potatoes and apples. Transactions of the ASAE 38 (5), 15551561. Varghese, Z., Morrow, C.T., Heinemann, P.H., Sommer, H.J., Tao, Y., Crassweller, R.W., 1991. Automated inspection of Golden Delicious apples using color computer vision. American Society of Agricultural Engineers 91 (7002), 16. Zhang, M., Ludas, L.I., Morgan, M., Krutz, G.W., Precetti, C.J., Zhang, M., Meyer, G.E., DeShazer, J.A., 1998. Applications of color machine vision in the agricultural and food industries. SPIE Proceedings Precision Agriculture and Biological Quality 3543, 208219.