The key problem for picking robots is to locate the picking points of fruit. A method based on the moment of inertia and symmetry of apples is proposed in this paper to locate the picking points of apples. Image pre-processing procedures, which are crucial to improving the accuracy of the location, were carried out to remove noise and smooth the edges of apples. The moment of inertia method has the disadvantage of high computational complexity, which should be solved, so convex hull was used to improve this problem. To verify the validity of this algorithm, a test was conducted using four types of apple images containing 107 apple targets. These images were single and unblocked apple images, single and blocked apple images, images containing adjacent apples, and apples in panoramas. The root mean square error values of these four types of apple images were 6.3, 15.0, 21.6 and 18.4, respectively, and the average location errors were 4.9°, 10.2°, 16.3° and 13.8°, respectively. Furthermore, the improved algorithm was effective in terms of average runtime, with 3.7 ms and 9.2 ms for single and unblocked and single and blocked apple images, respectively. For the other two types of apple images, the runtime was determined by the number of apples and blocked apples contained in the images. The results showed that the improved algorithm could extract symmetry axes and locate the picking points of apples more efficiently. In conclusion, the improved algorithm is feasible for extracting symmetry axes and locating the picking points of apples.

This work has two supplementary tables and four supplementary figures that do not appear in the printed article but that accompany the paper online.

The fruit picking robot is one of the main research directions of modern agricultural development. The key problem is how to realize the goal of identification and localization of fruit through accurate use of machine vision technology. Under natural conditions, fruit have different growth postures due to different soil, seasons and weather conditions, which have strong impacts on locating the picking points of apples and the follow-up picking tasks.

Several studies have been carried out to locate fruit under natural conditions (

As seen from the literature above, these methods have some unsolved disadvantages, such as low accuracy rate, highly time consuming, complexity of the operating process, etc. These disadvantages, to some extent, restrict the real-time capability of the apple harvesting robot in natural scenes. Thus, a new method should be proposed to solve these problems.

Symmetry axes detection of a 2D point set based on convex hull was presented by

Based on the descriptions above, this paper will try to describe a convex hull theory and contour symmetry axes extraction algorithm-based method to locate the picking points of apples. Firstly, the image is transformed from RGB color space to L*a*b* color space, and then the K-means color clustering algorithm is used to detect apples. Secondly, to weaken the influence of noise on the extraction of symmetry axes, image pre-processing algorithms, including mathematical morphology and noise removal, are carried out to remove noise and smooth the edges of the apples. Then the convex hull is used to replace the apple’s contour, which is useful to decrease the computational complexity. Finally, a moment of inertia algorithm is utilized to extract the symmetry axes of apples, which are used to realize the purpose of locating the picking points of apples accurately.

A personal computer with a 2.60 GHz processor and 4.0 GB of RAM was used as the hardware component of the computer vision system, and all algorithms were developed in MATLAB vers. R2013a. A digital camera (Fuji film A900, CMOS color camera) was selected, and the shooting distance was approximately 1.5 m. Our research will focus on the ‘Fuji’ apple (

The picking point is located on the peduncle of an apple, and it is one of the intersections of the symmetry axes and the contour of the apple. The intersection near the peduncle of the apple is the picking point. Searching for the picking point of an apple target is one of the most important tasks for the apple picking robot. After detection of the picking point, the robot can drive the end effector to cut the peduncle at the picking point and then fulfill the picking task for the apple. Through this manner of shearing, damage and decay of apples can be effectively decreased during storage processes and transportation, thus reducing loss effectively. The picking point, convex hull, and contour of an apple are shown in

Apple contour extraction is one of the most important steps in locating apple targets. To extract the contours of apples accurately, the K-means clustering algorithm was used to extract the regions of apples, and then the vertexes of the convex hull of each apple were extracted, which were used to replace the contour points of each apple and were useful for improving the operation speed.

There are no simple formulas for conversion between RGB and L*a*b*. The L*a*b* color space is based on the XYZ color space. Thus, RGB should be converted to the XYZ color space first and then transformed into the L*a*b* color space. The formulas that transform the RGB color space into the XYZ color space can be expressed as,

The formulas that transform the XYZ color space into the L*a*b* color space are as follows,

where the function is,

The K-means clustering algorithm can be employed to cluster the apple images into several different classifications based on the a* and b* color components, regardless of the brightness.

The K-means algorithm is an unsupervised learning algorithm that groups data objects into several clusters to obtain the highest similarity between objects in the same cluster but the minimum similarity between objects in different clusters. That is, single pieces of data are segmented into specified clusters through iterative search (

The K-means clustering algorithm is described as follows,

_{1}(1), _{2}(1), …, _{k}(1), where

_{j}(_{i}(_{j}(

_{j}(

_{j}(_{j}(

For the K-means clustering algorithm, the parameter

The results of K-means clustering are shown in

The number of peripheral points in

Based on the good symmetry characteristics of apples, the symmetry axes of apples can be identified as a way to locate apple targets. There are many algorithms for target symmetry axes extraction, such as the Euclidean distance algorithm (

The moment of inertia of the curve _{1} =

Specifically, _{1} =

Assuming that the function

or:

Two extreme values of the moment of inertia can be obtained. The line is a symmetry axis if it makes the moment of inertia reach one of its extreme values (maximum value or minimum value, depending on the shape of the curve); the line that reaches another extreme value (minimum value or maximum value) of the moment of inertia is perpendicular to the symmetry axis of the curve.

Because the moment of inertia is invariant with shifts and rotations of the coordinate axis, the extraction of symmetry axes of any symmetrical curve in any placement can be realized by locating the line when the corresponding moment of inertia reaches its extreme value. As shown in

To verify the validity of the improved contour symmetry axes method presented in this paper, four types of images containing 107 apple targets were used to conduct the experiment. The test was run on 30 single and unblocked apple images, 20 single and blocked apple images, 5 images containing 12 adjacent apples and 45 apples captured in 6 panoramas. The performances of the proposed methods of the first two types of images were compared to those of the unimproved method and the method of principal inertia axis (

where _{11}, _{20}, _{02} are two-order center moments of the binary image. They can be calculated by Eq.

where

From the equation above,

The location results of the last two types of apple images were not compared to the unimproved method and PIA method. There were many apple targets in the images, and some adjacent apples occluded severely, which resulted in the real contour of the apple being unable to be extracted for the follow-up experiment.

The locations of real symmetry axes were found through observation, and the location errors were the angles between the obtained symmetry axes and the real symmetry axes. Here, we assumed the angle of the real symmetry axes was

RMSE (root mean square error) was used in this paper to measure the performance of the method proposed. RMSE can be used to judge the deviations between the true value and the value obtained from the test. The picking point is on the symmetry axes of the apple, so the location error of the picking point is the location error of the symmetry axes of the apple. The picking point is one of the intersections of the symmetry axes and the contour of the apple. In this paper, the convex hull was used to replace the contour. Thus, the picking point is defined as one of the intersections of the symmetry axes and the convex hull, and it should be determined by the variety of the apple.

For 30 single and unblocked apple images, the processing step was relatively easy. The main procedures of the experiment were as follows,

The location results of apple targets are shown in

In natural scenes, the localization of apples is always influenced by being blocked by branches and leaves, light intensity, shadows on the surface of apples and the ripeness of apples. Twenty single and blocked apple targets containing these four situations were selected to conduct the experiment. The main procedures were the same as above. The results of the location of single and blocked apples are shown in

Some apples in panoramas and images containing adjacent apples required reconstructing the contour for occluded areas that were too large. The main procedures for this were as follows,

The experimental results and experimental data of images containing adjacent apples are shown in Suppl. Fig. S3 and Suppl. Table S1 [pdfs online], and those of the panoramas are shown in Suppl. Fig. S4 and Suppl. Table S2 [pdfs online]. As clearly shown in Suppl. Tables S1 and S2, the RMSE values of images containing adjacent apples and panoramas were 21.6 and 18.4, respectively, and the average location errors were 16.3° and 13.8°, respectively. The runtimes of these two types of images were determined by the number of apples and that of blocked apples contained in the image. From Suppl. Figs. S3 and S4, we can see that all of the apples in the images can be found and the picking points of the apples could be extracted.

From

Errors in location results (see

Although the location error of the presented algorithm is obvious for some apple targets, the location error of these apple targets using the proposed algorithm is lower than that of the PIA method and the unimproved algorithm.

A comparison of the performance of the PIA method, the unimproved method and the presented method is shown in

For images containing adjacent apples, the entire contour of some apples could not be extracted because they were being blocked by adjacent apples or leaves, and thus, the contour of these apples should be reconstructed. Though there were location errors of reconstruction, the apple targets could be located. Location errors of some unblocked apple targets were larger (targets 3 and 11 in Suppl. Fig. S3 [pdf online]). This was due to the poor symmetry of these apples. For apples captured in panoramas, all apples in the images could be found. It was precise in locating some of the apples. The smallest location error was 0.0°. There were also some apples with larger location errors: the largest location error was 38.1°. Some of them were because of poor symmetry. Others were because the convex hull of the apple could not express the region of the apple accurately, which resulted from branches and background factors, light intensity, being blocked by leaves and/or the maturity of the apple. Details are shown in Suppl. Fig. S4 [pdf online].

For these two types of apple images, the runtime was slightly longer; this resulted from the execution loop of every apple and the reconstruction of the blocked region of the blocked apples in the images.

In summary, this paper utilized the convex hull theory in combination with the K-means clustering algorithm and mathematical morphology to replace the contour curve of single and unblocked apple targets with the convex hull, thereby improving the precision of the target localization, enhancing the efficiency of operation, and simplifying the computational complexity of the original algorithm. The moment of inertia method was used to extract symmetry axes of apple targets, which could better locate the picking points of apples. The correctness and the effectiveness of the algorithm presented in this paper showed that the improved algorithm was feasible for extracting symmetry axes and locating the picking points of apples. For occluded apple targets, though the apple targets could be located, the location result was not accurate enough because they were blocked by leaves, stems or other apples. Therefore, further study is needed to improve the accuracy of the location of occluded apple targets.

The authors would like to thank Leilei Niu (graduate student, College of Mechanical and Electronic Engineering, Northwest A&F University) and anonymous referees for their helpful comments and suggestions.