Dynamic segmentation to estimate vine vigor from ground images

The geographic information required to implement precision viticulture applications in real fields has led to the extensive use of remote sensing and airborne imagery. While advantageous because they cover large areas and provide diverse radiometric data, they are unreachable to most of medium-size Spanish growers who cannot afford such image sourcing. This research develops a new methodology to generate globally-referenced vigor maps in vineyards from ground images taken with a camera mounted on a conventional tractor. This monocular camera was able to sense in the visible, NIR, and UV spectra, selectively isolated with bandpass filters. The versatility of the system was further enhanced by implementing two sampling levels: intensive coverage of 1 m2 and super-intensive for 0.1 m2. The core of the procedure resides in the algorithm for automatically segmenting the filtered images in such a way that relative differences in canopy vigor were objectively quantified. The calculation of the dynamic threshold involved the mathematical concepts of gradient and curvature. Field results showed that relative differences in vine vigor can be detected from NIR-filtered images and intensive sampling. Furthermore, individual images were successfully merged into a global vigor map that can be directly employed by end-users. Super-intensive sampling and UV perception were not appropriate for building vigor maps, but could be of interest for other agronomical purposes as the early detection of diseases. Field tests proved the feasibility of building global vigor maps from ground-based imagery, and showed the potential of this technique as a predictive instrument for modest-size producers. Additional key words: dynamic threshold; GPS; NIR; precision viticulture; UV; vigor map.

Palabras clave adicionales: GPS; mapas de vigor; NIR; umbralizado dinámico; UV; viticultura de precisión.Dynamic segmentation for vine vigor estimation practical for European vineyards.Illumination problems in outdoor environments have always been very difficult to handle and, despite passive sensors have been widely used over the years to measure light reflectance, active sensors seem to bring certain advantages for field applications (Kim & Reid, 2007).Many times, the difficulties found in image segmentation come from unpredictable illumination changes in the field.Some authors have tried to solve this problem with texture analysis (Nuske et al., 2011) or through spatial filters (Ramalingam et al., 2003).There are some general purpose segmentation methods (Otsu, 1979), but unfortunately they are not efficient when applied in open fields due to always varying lighting conditions.However, there exist commercial devices that have counterweighted these illumination challenges with active lighting as, for example, CropCircle TM (Holland Scientific Inc., Lincoln, NE, USA) and Greenseeker TM (NTech Industries Inc., Ukiah, CA, USA).The latter, in particular, has been used in vineyard fields to sense canopy reflectance and calculate vegetation indices, which have been further related to canopy porosity (Tardáguila et al., 2008), leaf area index projected onto a vertical plane (VLAI) (Drissi et al., 2009), and pruning weight (Stamatiadis et al., 2006).Mazzeto et al. (2009) corroborated the feasibility of calculating standard vegetation indices such as the NDVI with Greenseeker TM in vineyards; nevertheless, these commercial devices are not polyvalent and the information obtained from them has seldom taken part in decision making algorithms of PA applications.Some of them can only be used in bulk crops or do not incorporate GPS receivers, which complicates the sharing of data coming from other sensors with the final goal of explaining variability and casting predictions.
The ultraviolet (UV) band has not been as profusely studied as the infrared band due to the high reflectance of vegetation in the infrared.In addition, the atmospheric dispersion has a strong attenuation effect in the ultraviolet band, which limits this spectral band for sensing vegetation remotely (Noble & Crowe, 2005).Despite these drawbacks, there have been several studies focused on UV perception.Verhoeven & Schmitt

Introduction
The agricultural sector of industrialized countries is currently facing up a crisis that requires the urgent introduction of structural changes in crop production; the sort of changes that only technology is able to make.The set of new methodologies adopted by Precision Agriculture (PA) results in the practical application of information technology (IT) to agricultural production.In line with this philosophy, this research proposes an algorithm for the automatic segmentation of images used in a ground-based and non-invasive system to generate globally-referenced vigor maps.The goal of these maps is tracking the relative variation of vegetation, with independence of external data sourcing coming from satellites or aerial vehicles, in order to estimate vine yield and wine quality before harvesting time.
Light reflectance from crops, canopy, and fruits, has been widely studied for several bands of the electromagnetic spectrum.Gausman (1977) found that the highest percentage reflected by vegetation is located in the infrared band, and Weekley (2007) also confirmed the existence of a specific range of that band where the percentage of reflectance grows dramatically, generally known as the ''red edge'' and located between 700 nm and 800 nm.Traditionally, the predominant technique used to sense physiological properties of leaves and fruits has been spectrometry, a manual procedure capable of covering several bands of the spectrum (Tucker, 1979;Peñuelas et al., 1994).The -relatively-new technique of remote sensing uses artificial satellites and airborne imagery for data acquisition, offering an attractive alternative to conventional methods, typically manual and very often invasive.Remote sensing applications for viticulture tend to search in the infrared band, either in the near infrared (NIR) or in the thermal infrared band of the spectrum (Best et al., 2011).Reflectance in the NIR has been applied to the early detection of stress in plants (Chaerle & Van Der Straeten, 2000), to monitor growth and health of vegetation (Weekley, 2007), to detect nitrogen deficiency in crops (Noh et al., 2005), or to predict the content of sugar in fruits (Lu & Ariana, 2002).Praat et al. (2004) built a mechanical structure with a flat panel to capture the side view of vine canopies under a controlled background.They counted green vine pixels on the magenta background panel to estimate the biomass.The system withstood sunlight problems, but the bulkiness of the solution makes it highly un-ied to the individual level of a leaf or fruit.Canopies have the same characteristics of each leaf, but such properties change as the surface of leaves acts, many times, as a polarizing filter, reflecting light to different directions.Due to this fact, image properties very often depend on the position of the camera.
In wine vines, it is essential to detect the best time for harvesting.It is not rare that vines from the same field have to be harvested in different periods, as they typically do not mature at the same time.Differential harvesting is justified when producers are searching for a wine with very special characteristics (Best et al., 2011).A practical way to carry out differential harvesting can be by creating field maps that help to harvest the grapevines when their phenolic maturity is optimal (Lamb et al., 2004).According to Johnson et al. (2003) leaf area may be related to vegetation vigor, as well as to crop diseases and infestations, plant water stress, fruit characteristics, and wine quality.The most popular technologies for monitoring key parameters in agricultural and forestry production have used airborne techniques, which are not reachable for the majority of modest wine producers in Spain.More efficient alternatives can be feasible as long as external data sources are replaced by user-operated data acquisition systems.In this regard, this article proposes a novel procedure to endow conventional vehicles with a mapping system based on the fusion of GPS positioning information with a vision-based perception system, with the final objective of tracking relative differences in vegetative vigor, quantitatively expressed as percentage of vegetation and counted by the algorithm as number of vegetation pixels per image.This system is meant to be used by smallholders, who represent an important percentage of producers in Spain.

Material and methods
The acquisition system for mapping vigor during the field tests was mounted on a conventional tractor (JD 5820, Deere & Co., Moline, IL, USA).This vehicle is equipped with the StarFire iTC GPS receiver used in the JD GreenStar system, capable of delivering free signals SF1 with static accuracy of 75 cm and pass to pass accuracy of ± 33 cm, as well as licensed signals SF2 with static accuracy of 25 cm and pass to pass accuracy of ± 10 cm.Although the SF2 signal can be activated anytime by paying a subscription fee, the architecture proposed only took into account free sig-nals in order to make the system affordable to a wider range of producers.The principal sensor for visual perception was a CCD camera (JAI, Copenhagen, Denmark).This camera is monocular, monochrome, sensitive in the range UV -NIR, and has a pixel depth of 8 bits per pixel.Image acquisition and storage was carried out with a laptop computer (2.20 GHz and 1.5 GB of RAM, Fujitsu Siemens Computers, Tokyo, Japan) located inside the cabin and connected to the camera via Ethernet.In order to assure the continuous supply of electric power to the camera, computer, and GPS screen, the tractor featured a secondary battery of 12 volts connected in parallel with the main engine battery.The selection of a particular field of view (FOV) and spectral band required the coupling of a lens of either 8 mm or 25 mm of focal length (Goyo Optical Inc., Saitama, Japan), in combination with an optical filter (Midwest Optical Systems Inc., Palatine, IL, USA).Two pass-band filters were employed over the experiments, one centered at 324 nm in the ultraviolet band (UV-A and UV-B, from 270 nm to 375 nm), and the other centered in the NIR at 880 nm, with an operating range between 840 nm and 1,100 nm.Incidentally, the UV filter also included a narrow band of low transmission in the NIR from 690 nm to 750 nm.
Field tests were carried out in July and August, 2010, in the vineyards belonging to a winery located near Requena (Valencia, Spain).All the plants mapped with the tractor were 20-year-old Cabernet-Sauvignon vines, approximately oriented along the East-West direction.The field had an average elevation of 650 m above sea level.Inter-row lanes were 3 m wide and about 130 m long.Different camera positions were tried over the tests, but the best alternative turned out to be ahead the tractor front, just above the front ballast as shown in the schematic of Fig. 1.Table 1 lists the combination of lenses and filters implemented in the vehicle to perform the field tests.

Image analysis (I): Preliminary inspection of the raw images
The first step of the image analysis consisted of superposing all the histograms of the set of images belonging to each pass listed in Table 1.On the whole, nine passes were examined and the initial assessment, based on the morphology of the histograms overlaid for each pass, revealed a dominant accumulation of pixels for two distinct levels of gray when sensing in the NIR spectral band.This outcome was favored by limiting the light reflected from the scene -and captured by the sensor-to the NIR, which tends to stand out vegetation from the background of an image.The accumulated plot of the histograms demonstrated that most of the pixels usually concentrated around two definite centers of mass, but these centers varied depending on the mean gray level of the images.Additionally, there were also local variations in the position of these centers among images coming from the same pass.Therefore, the differences among passes and among images of the same pass resulted in the need of a customized dynamic threshold for isolating vigorous leaves; that is, a different threshold for each image adapted to its particular characteristics.The application of a real-time dynamic threshold to each single image allowed the straightforward estimation of the percentage of vegetation for a given sampled area.

Image analysis (II): Dynamic segmentation for estimating plant vigor
The series of histograms justifies the benefit of a specific threshold per image if accurate vegetation percentages need to be estimated.However, finding each threshold directly from the histograms is not an easy task, and can be detrimental for real time implementations where image analysis cannot extend in excess.To overcome this time-consuming task, critical information can be managed more efficiently through the segmentation profile plot of Fig. 2b, where the abscissa axis represents the 8-bit grey level -or intensity level -analyzed and the ordinate axis provides the percentage of pixels in the image whose intensity level is higher than that analyzed.Fig. 2a is a sample image of the NIR-filtered images used in this study.Note, for example, that for a grey level of 100, Fig 2b indicates that approximately 90% of the pixels in the image of Fig. 2a have an intensity value above 100.Logically, the plot states that 100% of the pixels have a gray level above 0, and similarly, 0% of the pixels have a grey level higher than 255.Obviously, this fact about the boundary values of segmentation profiles is common to all the images, but what results most interesting for the efficient execution of the algorithm proposed is the actual profile of the curve.Generally speaking, these curves will be S-shaped, being the particular form of the profile what holds the critical information for selecting the threshold.Given that the object of interest -vegetation of vines-has been enhanced with the NIR filter, a sharp drop for the segmentation profile is expected in the vicinity of the optimal threshold.In practical terms, as soon as the checked gray level crosses the optimal threshold, a significant amount of pixels in the pattern plot change their condition from vegetation pixels to non-vegetation pixels.This steep change in the profile of Fig. 2b occurs around a threshold of 100.
The segmentation profile plot (Fig. 2b) provides the key information to find the best threshold; what remains is the search of the specific point of the profile after which the most drastic drop starts.This calculation must be fast, automatic, and as reliable as possible, given that correct thresholds will result in trustable vegetation percentages.With the purpose of finding the optimal threshold for each image, the mathematical concepts of gradient (first order variation) and curvature (second order variation) were applied to the contour given by segmentation profile plots.The underlying idea is the automatic detection of the largest change in ordinate for a constant change in abscissa (intensity value).A direct way to locate and quantize big sets of pixels changing their classification status when the applied threshold varies in the range [0, 255] is through the concept of gradient (i), whose mathematical expression for discrete elements is given in Eq. [1], where the grey level considered is represented by i, the finite interval of grey levels -i.e. the resolution-is δ, and P(i) is the percentage of pixels whose grey level is above the threshold i.
Apart from a significant drop in the profile of Fig. 2b, a change in the curvature of the plot is also noticeable in the neighborhood of the downwards fall; therefore, the calculation of the curvature for gray levels in the vicinity of the drop should lead to high values as well.As a matter of fact, field experience showed that the optimal gray levels were obtained by averaging the grey level of maximum gradient with that of maximum curvature and then shifting the value back by δ pixels.Curvature is a second order calculation that indicates gradient variations for a discrete jump δ.The second derivative is usually taken as an approximation of the curvature, which can be applied to segmentation profile plots according to Eq. [2], where (i) is the gradient for grey level i, δ is the interval of grey levels considered, and 2 (i) is the curvature for intensity level i.
Once the highest values for the gradient and curvature have been found and related to their corresponding gray levels i max ∇ and i i max max ∇ ∇ 2 , the dynamic value of the i max ∇ threshold μ can be estimated according to Eq. [3].A grey level interval δ = 5 was chosen throughout the entire analysis of the images acquired in 2010.
The final operation for each individual image of the series consists of the estimation of the vegetation percentage, which is straightforward after the calculation of the most favorable threshold μ.Recall that every image generally resulted in a different μ, as shown in the plot of Fig. 3a.The move from the threshold plot (Fig. 3a) to the final vegetation percentage (Fig. 3b) for any series of images simply entails the segmentation of each image with its particular dynamic threshold μ, so that those pixels with a gray level superior to μ are classified as vegetation.The total number of pixels classified as vigorous vegetation divided by the resolution of the image analyzed gives the percentage of vegetation for that image.This is the parameter to be tracked and inserted in the vigor map.
All the operations applied to every image in order to estimate the final percentage of vegetation can be  summarized in the schematic of Figure 4, which provides a block diagram of the algorithm with the sequence of steps that need to be taken from the initial acquisition of the image to the final construction of the vigor map.In addition to performing the calculations described above in Eqs.
[1] to [3], the algorithm also stores the vital information necessary to compose the vigor map.In particular, a text file collects the dynamic thresholds and percentage of vegetation for each image processed.The combination of this perceptive information with the global position at which each image was taken is essential to assemble the vigor map.Further details on how vigor maps were represented with conventional software for the passes listed in Table 1 are included in the next section.

Results
Before attempting to build vigor maps following the procedure outlined in Fig. 4, it is necessary to check the performance of the segmentation routine, as dynamic thresholding is the most delicate stage of the mapping algorithm.The text files output by the algorithm facilitated this validation because every image analyzed was always associated to an optimized dynamic threshold μ.Fig. 5 illustrates how the algorithm segmented NIRfiltered images by separating vegetation from the background, typically represented by sky, trellis wires and posts, or vine trunks (stocks).The algorithm found correct thresholds in the majority (≈ 97%) of the NIR images, but encountered difficulties in the UV band, mainly caused by the confusing reflection of the sky.
Several operations need to be successfully accomplished in order to obtain correct vegetation percentages and, thus, useful vigor maps.First, an acceptable level of perception has to be assured, which implies selecting the most favorable spectral band and identify-  Second, the geographic localization of every image in a global reference frame is essential to generate usable maps.Third, an aspect of great importance with regards to vigor maps is the sampling-capacity of each map, determined by the user when selecting the lenses of the camera.Fig. 6a,b represent a vigor map of the same row; however, the size of the area sampled is tenfold for the former.The area perceived by the 8 mm lens of the map drawn in Fig. 6a seems more informative than that obtained with the 25-mm lens of Fig. 6b.Although Fig. 6a takes more area than Fig. 6b, the latter has, in contrast, the added benefit of detecting details not available with larger images, such as fruit properties or infestations.

Discussion
The spectral band that showed the best capacity to highlight the object of interest in this research (vegetation) was the near infrared.The configuration of the system proposed demonstrated the capability of generating vigor maps with NIR vision.Despite a few publications related to the detection of vegetation in the UV band (Hahn, 2009;Verhoeven & Schmitt, 2010), its applicability for the objectives set and the nature of the environments tested did not meet the expected requirements.In particular, unhealthy vines affected by downey mildew (Plasmopara viticola) and stained on the leaves by oily spots were not detected by the UV-based vision system, leaving this research topic completely opened.
The developed algorithm mapped the trajectory of the vehicle and acquired all the images with the computer onboard running a customized program.Although vigor maps were successfully generated, data synchronization was difficult at some points, and consequently, the tight integration of both algorithms -localization and perception-still remains to be done.According to the goals initially set for the creation and use of vigor maps, intensive sampling turned out to be more appropriate than super-intensive sampling, as the field of view covered by the 8 mm lens provided the best window to estimate vegetation changes along the rows.Super-intensive sampling (Fig. 6b), on the contrary, may be suitable for other purposes such as the early detection of vine diseases.
The validation of the entire system requires a comparison between the information stored in the map and the actual assessment of the data in the field.This final step in the validation of the method will require data from several seasons, and therefore was not possible for the timeframe within which the experiments took place, although   is one of the necessary tasks for upgrading the system in the near future.Field data from several harvesting operations will be necessary to properly correlate prospective yield maps with reality and eventually make useful predictions.The percentage of vegetation estimated in Figs.6a and 6b give a sense of the vigor of the vines, but field validation has to establish the numerical equivalence to what Hall et al. (2002) denominate high vigor, a principal contributor of high yield, delay in the maturity of fruits, and a medium-low quality of the produced wines.This research proposes a mapping system that approximates emerging technology to medium-small wine producers.The methodology developed combines global positioning information from a GPS receiver with NIR ground-based imaging to create vigor maps of vines.All the necessary equipment was implemented in a standard tractor, and the perceptive capacity of the system can easily be expanded by enlarging the sensitivity of the spectral band of the imaging sensor.Preliminary results showed the viability and potential of this ground-based system, which provides a new approach in processing the images and obtaining grapevine vigor maps.The ultraviolet reflectance of vegetation was also explored as a potential indicator for plant vigor, but the calculation of a dynamic threshold becomes more complex.Nevertheless, UV may bring further information to detect other properties of the canopy, as may do other bands of the spectrum such as thermal infrared or thin slices in the visible.Finally, the same system architecture can be applied to alternative perception sensors to systematically obtain comparable maps where, not only the type of radiation, but also the sample size would be adjusted by the growers to their specific needs.

Figure 1 .
Figure 1.Front view (left) and top view (right) of the camera position respect to the vehicle and vine rows.

Figure 2 .
Figure 2. Original image with 256 levels of gray (a), and its corresponding behavior pattern (b) showing its segmentation profile curve and optimal threshold value (vertical line).
Dynamic segmentation for vine vigor estimation

Figure 3 .
Figure 3. Optimal threshold µ plot (a) showing one threshold value per image, and percentage of vegetation coverage in each image (b) for a pass of 19 images.

Figure 4 .
Figure 4. Block diagram of the proposed algorithm to create ground-based vigor maps.

Figure 5 .
Figure 5. Resulting image of applying the dynamic thresholding algorithm to NIR-filtered images.Red color shows non-vegetation pixels.

Figure 6 .
Figure 6.Intensive-sampling (a) and super-intensive-sampling (b) vigor map.Intense red color shows tractor path when capturing images for pass 2 (a) and for pass 6 (b).
Dynamic segmentation for vine vigor estimation

Table 1 .
Summary of field tests selected to map vine vigor