Computer vision applied to saffron flower ( Crocus sativus L . ) processing

This paper presents a computer vision system to obtain, using image analysis, the optimal cutting point of saffron flowers in order to obtain their stigmas. For this purpose, an effective and flexible computer program has been developed to process the flower image in order to obtain the cutting point to be sent to the cutting element. Furthermore, experimentation with real saffron flowers has been carried out in order to validate the developed application. In particular, the tests show that the method has good robustness and high success percentage in the flower characterization regardless the shape and size of the flower. The high image processing rate of the proposed method (20 computations s–1) would allow to greatly increase the production rate obtained with an automated cutting system compared to that obtained with the traditional hand method. Additional key words: automation; flowers processing; vision system.

Saffron is a spice that has been well known since antiquity.It is derived from the dried stigma of the flower of the saffron crocus (Crocus sativus L.), a species of crocus in the family Iridaceae.The flower has three stigmas, which are the distal ends of the plant's carpels.The stigmas together with the style and the stalk or peduncle connecting the stigmas to the rest of the plant are often dried and used in cooking as a seasoning and colouring agent.Saffron has for decades been the world's most expensive spice by weight.It is characterized by a bitter taste and an iodoform or haylike fragrance and it is a much-sought ingredient in many foods worldwide.Saffron also has medicinal applications (Premkumar et al., 2006).
Today, saffron is cultivated from the Western Mediterranean (Spain) to India (Kashmir).Spain and Iran have been the largest producers in the recent past years, accounting together for more than 80% of the world's production.Nowadays, Iran ranks first in the world production of saffron with more than 94% of the world yield.Annually, around 300 tones of saffron are produced worldwide (ICEX/Estacom database, http://www.icex.es).In Europe, saffron production is almost limited to the Mediterranean.Short communication.Computer vision applied to saffron flower (Crocus sativus L.) processing which are separated each other.The characterization of the transition section is critical to cut and released the desired stigmas.The petals range from 34 mm to 42 mm in length with a mean of 40 mm (twenty flowers were analyzed).The three flower stamens are joined with the base of three non-consecutive petals.They are round 16 mm in length; this includes the anther (approx.6 mm).From the peduncle base arises the style, which goes outside the flower rounded by the corolla tube.The style exceeds the transition section for 4-8 mm, and then divides into three stigmas that contain the saffron.Each stigma is 25-30 mm in length and usually hangs between the petals of the open flowers.
In order to obtain the greatest quantity of spice with good quality, the three stigmas and a small portion of the style must be cut together.Note that the style becomes firstly yellow and then white in the opposite part of the stigmas, which means a loss of spice quantity.In short, the cut should be made in a range of ±2 mm to the transition section, which is located at the flower base, where it gets darker and thicker.With this approach, the three stigmas are released along with a style portion ranging from 2 mm to 10 mm.If the cut, for some reason, is made higher, the stigmas would appear separated, Once harvested, the saffron flowers are taken to an inside location and emptied onto a table for the ''monda'', i.e. the manual act of peeling or stripping the stigmas from the flowers.The whole operation takes about 4 s flower -1 and is mainly responsible for the high cost of saffron.Upon extraction, stigmas are quickly dried and toasted, preferably sealed in airtight containers.One kilogram of dry saffron requires between 110 and 165 thousands of flowers (around 80 kg of flowers).
Nowadays, almost all the saffron manipulations are carried out manually.In fact, today there is no real effective automated procedure to obtain the stigmas of the saffron flower.For this purpose, an automated cutting system could be designed with the following subsystems: (a) flower transporter, (b) computer vision system, (c) automatic cutting element, and (d) stigma separation system.This paper is focused on subsystem (b), i.e., the aim of this work is to develop a computer vision system to obtain the optimal cutting point of saffron flowers to be sent to an automatic cutting element.Subsequently, the stigmas could be separated from the petals with an air flow, because their aerodynamic resistances are substantially different.
Computer vision is becoming more and more used in industry for fruits (Blasco et al., 2008;Rakun et al., 2011), vegetables (Story et al., 2010) or even both (Rocha et al., 2010).These related works are focused from inspection (Story et al., 2010) to manipulation (Huang and Lee, 2010) either outdoors in the field (Swain et al., 2010), inside a greenhouse (Story et al., 2010) or in a research laboratory (Omid et al., 2010).Developed countries focus their work on reducing human manipulation, automating these tasks (e.g.fruit/ vegetable classification) as much as possible (Rocha et al., 2010;Unay et al., 2011) and increasing the product quality using some kind of anticipation (Story et al., 2010).For this purpose, commercial robots (Huang and Lee, 2010) and mechanisms specially designed for the particular application (Story et al., 2010;Swain et al., 2010) are used together with image processing of color (Rocha et al., 2010;Swain et al., 2010) or monochrome images (Huang and Lee, 2010).Furthermore, highlevel programming languages like Delphi or C++ (Huang and Lee, 2010;Swain et al., 2010) and vision libraries like Matrox Image Library (Huang and Lee, 2010) are also used.
Next, the geometry of the saffron flower is described.The purplish corolla of the saffron flower has a dark tube-shape base next to the peduncle, as shown in Fig. 1a.The light part of the corolla is the six petals, C. Perez-Vidal et al. / Span J Agric Res (2011) 9(4), 1176-11811178 which implies a loss of spice quantity.On the contrary, if the cut is made lower, the style portion would be bigger, which implies a loss of spice quality, besides that the stigma separation from the corolla tube would require a stronger air flow or other mechanical actions.
The proposed method consists of moving the saffron flowers one by one in front of a vision system or camera using a transport system (the flowers are placed manually on the transporter by the operators, see Fig. 2).The image acquired by the camera is sent to an industrial computer, where it is analyzed in order to obtain the cutting point (with the cutting plane perpendicular to the flower axis) to properly separate the three stigmas along with the style from the rest of the floral elements.This information is sent from the computer to the cutting element or cutting head.
The camera to be used is monochrome (no colorbased analysis is performed) since a black background contrasts sharply with the light saffron flower.The optical axis of the camera (Fu et al., 1987) is perpendicular and close (for a smaller machine size) to the flower axis.The following image processing steps are applied in a computer to the acquired image of the flower (see Fig. 3a): binarization, opening and closing, erosion, maximum/minimum point and offset.Next, these processing steps are detailed.
The binarization is a typical image processing operation (Niblack, 1986) that converts an image of gray levels (typically 256) to a black and white image.This operation is performed by comparing the value of each pixel in the image to a specified threshold value.If the pixel value is above the threshold value, the pixel is set to the maximum value in the image; otherwise it is set to 0. The threshold value has to be properly selected to correctly separate the saffron flower (object to be detected) from the background.After binarization operation, the original image acquired by the vision system is transformed into a binarized image with noise, see Fig. 3a.This noise is typically due to electromagnetic interference, a bad illumination, etc.
The opening and closing operations serve in image processing as a basic workhorse of morphological noise removal (Dougherty, 1992).Opening removes small objects from the foreground of an image, placing them in the background, while closing removes small holes in the foreground, changing small islands of background   (2) (4) Short communication.Computer vision applied to saffron flower (Crocus sativus L.) processing into foreground.In this work a disc-shaped structuring element is used for both operations.After opening and closing operations the binarized image with noise is transformed into the filtered binarized image (Fig. 3a).
The erosion is one of two fundamental operations (the other being dilation) in morphological image processing from which all other morphological operations are based (Dougherty, 1992).This operation is the key of the proposed image processing and is used to remove the flower peduncle, i.e. peninsulas of insufficient size, but keeping the flower corolla as unmodified as possible.In this work a square structuring element is used for this operation.After erosion operation, the filtered binarized image is transformed into the eroded or processed image (Fig. 3a).
The next step computes the maximum/minimum point (downward/upward flower) of the flower detected in the processed image.Finally, an offset is added/subtracted to the maximum/minimum point of the previous step in order to obtain the desired cutting point (see thick dashed line in Fig. 3a).This has two advantages: firstly, it allows to correct a repetitive error when applying the previous image processing; secondly, it allows to modify the desired quality of the resulting saffron.
The image processing steps described above could be also useful for the image processing of other types of flowers, or even for fruits and other vegetables (videos available at http://personales.upv.es/luigraca/SJAR/videos.htm).
The main features of the hardware used in this work for image processing are detailed below: -Computer: Industrial PC for rack mounting with Pentium ® 4 1.7GHz and 1GB of RAM equipped with Microsoft Windows XP Professional, Matrox Image Processing Library (MIL) 8.0 and Microsoft Visual C++ 6.0.-Camera (Sony XC-HR50): Monochrome; 1/3 type progressive scan CCD with square pixels; VGA resolution of 704×495 pixels at 60 full frames s -1 .-Lens: Special for machine vision with C-mount; focal length of 12 mm; maximum aperture ratio of 1:1.4; iris range from 1.4 to 16. -Frame grabber (Matrox Meteor II/MC): sampling rates up to 30 MHz; real-time transfer to system or VGA memory.-Illumination: an external enclosure has been used in this work to protect both the camera and the objects from the natural (or external) light.Inside the enclosure the illumination is controlled with fluorescent lamps (28W) with electronic ballasts and voltage regulators.
For the camera calibration, it has been used the direct linear transformation (DLT) method (Abdel-Aziz and Karara, 1971) at a known distance, where the image plane of the flower is located.In particular, the typical values of the pixel-to-world transformation for this research range from 0.1 to 0.25 mm pixel -1 .
The final application or program has been developed using Microsoft Visual C++ MFCs and has been registered with the name of SaffronCut®, which is currently being used experimentally by a Spanish saffron company (which is designing and developing the remaining subsystems of the automated cutting system in order to obtain an industrial prototype).The program was registered in 2009 with code V-26-09 in the Spanish Office of Software (Valencia).It processes the acquired image according to the steps described above in order to obtain the cutting point.The program has several configurable parameters.Some of them give flexibility (flower orientation: upward/downward, brightness of the background: dark(day)/light(day), direction of movement of the flower, etc.), other parameters are for the camera calibration and the remaining parameters modify the image processing settings as follows: -Gray-level threshold value: threshold from 0 to 255 (0 being black and 255 being white) used to binarize the acquired grayscale image.This value depends on the contrast, brightness, sharpness, etc. of the image.A typical value is 120.-Opening and closing values: radii in pixels of the disc-shaped structuring elements used to apply the opening and closing operations to the binarized image.These values depend on the amount of noise present in the binarized image.Typical values range from 2 to 8 pixels.-Erode value: width in pixels of the square structuring element used to erode the detected object (i.e.flower).This value depends on the width of the flower peduncle.A typical value is 15 pixels, which would eliminate a standard peduncle of 2 mm in diameter.-OffSet: number of pixels added/subtracted to the computed cut position.This is an important parameter for the final user since it allows to change the quality of the resulting spice: the value is increased to obtain a better quality to the detriment of spice quantity.This parameter depends also on the erode value.
The values of these image processing parameters have to be adjusted manually in the computer program by trial-and-error procedure depending on the environmental conditions: lighting, contrast, etc.Typically, each parameter value is successfully tuned after three attempts and, therefore, the tuning process takes very short time.
A processing example is shown in Fig. 3b for an upward flower, where the computed cutting point is shown as a horizontal line.It is worth mentioning that the developed program allows more than one saffron flower in the image plane.However, it is convenient to have at most one flower in the acquired image in order to allow a larger flower image, i.e. a better cutting resolution.Therefore, the minimum distance between flowers is the horizontal length of the image plane, i.e. 100 mm in this work.
The computer program processes the acquired image with one saffron flower in about 50 milliseconds.In other words, there are around 20 computations s -1 of the cut position.However, there is a limitation in the number of flowers that can be manually supplied to the flower transporter in the time unit.Moreover, the transporter and other mechanical parts of the system (e.g. the cutting element) must not be subjected to excessively high velocities to avoid undesired inertial effects, vibrations, etc.Therefore, it is estimated that the maximum flower processing rate consistent with these constraints is around two flowers per second, which is about eight times higher than that obtained with the traditional hand method (around 0.25 flowers s -1 ).For example, if the horizontal length of the image plane is 100 mm, the speed of the flower transporter could be up to 200 mm s -1 (2×100) and each flower would be processed at least ten times (20×100/200).
Preliminary tests of image processing were made in the laboratory with different types of artificial flowers in order to debug and validate the computer program.Subsequently, the evaluation of the proposed method was carried out at an actual plant using real saffron flowers during the flowering period.Obviously, the environmental conditions in the laboratory and plant were different but the image processing parameters of the program were easily adjusted for each case as mentioned above.
For the evaluation of the robustness of the proposed method, three basic types of saffron flowers were considered in the performance tests (Fig. 1b): closed, semi-closed and open, depending on their flowering date.A sample of 40 flowers of each type was ana-lyzed, which is representative due to the low variability in size, shape, color, consistency, etc. for each flower type.A flower is processed successfully if the cutting point obtained by the image processing system is located in a range of ±2 mm to the transition section (see Fig. 1a and the above description of the saffron flower geometry).The success percentage of the proposed image processing system for the optimal environmental conditions (lighting, contrast, etc.) was: 97.5% for closed (i.e., one of the 40 flowers was not successfully processed), 95% for semi-closed (i.e., two of the 40 flowers were not successfully processed) and 87.5% for open (i.e., five of the 40 flowers were not successfully processed).These results show that the proposed method has good robustness and a high success percentage in the flower processing regardless the shape and size of the flower.
Finally, the high image processing rate of the proposed method (20 computations s -1 ) would allow to greatly increase the production rate obtained with an automated cutting system, e.g., one person only averages about 1000 flowers h -1 with the traditional hand method.The implementation of the cutting element remains as further work.

Figure 1 .
Figure 1.Photographs of the saffron flower: (a) parts of the saffron flower (4 petals and 2 stamens have been taken away), (b) three basic types of flowers (from left to right), closed, semiclosed and open.

Figure 2 .
Figure 2. Examples of applications for flower cutting: (a) using a Cartesian robot, (b) using a pan/tilt robot.