Stem localization of sweet-pepper plants using the support wire as a visual cue

A robot arm should avoid collisions with the plant stem when it approaches a candidate sweet-pepper for harvesting. This study therefore aims at stem localization, a topic so far only studied under controlled lighting conditions. Objectives were to develop an algorithm capable of stem localization, using detection of the support wire that is twisted around the stem; to quantitatively evaluate performance of wire detection and stem localization under varying lighting conditions; to determine depth accuracy of stereo-vision under lab and greenhouse conditions. A single colour camera was mounted on a pneumatic slide to record image pairs with a small baseline of 1 cm. Artificial lighting was developed to mitigate disturbances caused by natural lighting conditions. An algorithm consisting of five steps was developed and includes novel components such as adaptive thresholding, use of support wires as a visual cue, use of object-based and 3D features and use of minimum expected stem distance. Wire detection rates (true-positive/scaled false-positive) were more favourable under moderate irradiance (94/5%) than under strong irradiance (74/26%). Error of stem localization was measured, in the horizontal plane, by Euclidean distance. Error was smaller for interpolated segments (0.8 cm), where a support wire was detected, than for extrapolated segments (1.5 cm), where a support wire was not detected. Error increased under strong irradiance. Accuracy of the stereo-vision system (±0.4 cm) met the requirements (±1 cm) in the lab, but not in the greenhouse (±4.5 cm) due to plant movement during recording. The algorithm is probably capable to construct a useful collision map for robotic harvesting, if the issue of inaccurate stereo-vision can be resolved by directions proposed for future work. This is the first study regarding stem localization under varying lighting conditions, and can be useful for future applications in crops that grow along a support wire.

[1]  H. Scharr,et al.  A stereo imaging system for measuring structural parameters of plant canopies. , 2007, Plant, cell & environment.

[2]  THEODORE P. PACHIDIS,et al.  Pseudo-Stereo Vision System: A Detailed Study , 2005, J. Intell. Robotic Syst..

[3]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[4]  J. Bontsema,et al.  An Autonomous Robot for Harvesting Cucumbers in Greenhouses , 2002, Auton. Robots.

[5]  Jizhong Deng,et al.  Extraction of Litchi Stem Based on Computer Vision under Natural Scene , 2011, 2011 International Conference on Computer Distributed Control and Intelligent Environmental Monitoring.

[6]  Yael Edan,et al.  Harvesting Robots for High‐value Crops: State‐of‐the‐art Review and Challenges Ahead , 2014, J. Field Robotics.

[7]  Ying Zhang,et al.  Research on trunk and branch recognition method of apple harvesting robot , 2012, Proceedings of 2012 International Conference on Measurement, Information and Control.

[8]  Julie Delon,et al.  Small Baseline Stereovision , 2007, Journal of Mathematical Imaging and Vision.

[9]  Pol Coppin,et al.  Reconstruction of tree structure from laser-scans and their use to predict physiological properties and processes in canopies , 2004 .

[10]  Rahul Sukthankar,et al.  Classification of plant structures from uncalibrated image sequences , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[11]  E. J. van Henten,et al.  An Autonomous Robot for De-leafing Cucumber Plants grown in a High-wire Cultivation System , 2006 .

[12]  Xiaoyan Hu,et al.  A Quantitative Evaluation of Confidence Measures for Stereo Vision , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  C. Glasbey,et al.  SPICY: towards automated phenotyping of large pepper plants in the greenhouse. , 2012, Functional plant biology : FPB.

[14]  Chris A. Glasbey,et al.  Combining Stereo and Time-of-Flight Images with Application to Automatic Plant Phenotyping , 2011, SCIA.

[15]  Raphael Linker,et al.  Determination of the number of green apples in RGB images recorded in orchards , 2012 .

[16]  Linda G. Shapiro,et al.  Computer and Robot Vision , 1991 .

[17]  J. C. Noordam,et al.  Automated rose cutting in greenhouses with 3D vision and robotics: analysis of 3D vision techniques for stem detection , 2005 .

[18]  J. Fripp,et al.  A novel mesh processing based technique for 3D plant analysis , 2012, BMC Plant Biology.

[19]  I. K. Sethi,et al.  Hierarchical Classifier Design Using Mutual Information , 1982, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  J. Bontsema,et al.  Mechanical weed control in sugar beet growing: the detection of a plant in a row. , 1991 .

[21]  E. J. van Henten,et al.  Pixel classification and post-processing of plant parts using multi-spectral images of sweet-pepper , 2013 .

[22]  Qiang Lu,et al.  Obstacle Recognition Using Multi-Spectral Imaging for Citrus Picking Robot , 2011, 2011 Third Pacific-Asia Conference on Circuits, Communications and System (PACCS).

[23]  E. J. van Henten,et al.  Robust pixel-based classification of obstacles for robotic harvesting of sweet-pepper , 2013 .

[24]  Marsha Jo Hannah,et al.  Computer matching of areas in stereo images. , 1974 .

[25]  Manuel A. Armada,et al.  Combination of RGB and Multispectral Imagery for Discrimination of Cabernet Sauvignon Grapevine Elements , 2013, Sensors.

[26]  Jake K. Aggarwal,et al.  Stochastic Analysis of Stereo Quantization Error , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..