Vision-based extraction of spatial information in grape clusters for harvesting robots

Grapes are likely to have collisions and be damaged by manipulations when harvesting grape clusters. To conduct an undamaged robotic harvesting, this paper focuses mainly on locating the spatial coordinates of the cutting points on a peduncle of grape clusters for the end-effector and determining the bounding volume of the grape clusters for the motion planner of the manipulator. A method for acquiring spatial information from grape clusters is presented based on binocular stereo vision. This method includes four steps: (1) calibrating the binocular cameras and rectifying the images, (2) detecting the cutting points on the peduncle and the centres of the grape berries, (3) extracting three-dimensional spatial coordinates of the points detected in step 2, and (4) calculating the bounding volume of the grape clusters. A total of 300 images were captured in the vineyard and were tested to validate the method for the cutting point detection, and the success rate was approximately 87%. The accuracy of the localisation of the cutting points was determined under outdoor conditions, and the accuracy in the Z and X directions was 12 mm and 9 mm, respectively. The acquired bounding volume of the grape cluster was compared with manual measurements, and errors in the height and maximum diameter were less than 17 mm and 19 mm, respectively. The elapsed time of the whole algorithm was less than 0.7 s. The demonstrated performance of this developed method indicated that it could be used on harvesting robots.

[1]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Yael Edan,et al.  Grape clusters and foliage detection algorithms for autonomous selective vineyard sprayer , 2010, Intell. Serv. Robotics.

[3]  Zhao Dean,et al.  System Design and Control of an Apple Harvesting Robot , 2020, ArXiv.

[4]  Salviano F. S. P. Soares,et al.  Automatic detection of bunches of grapes in natural environment from color images , 2012, J. Appl. Log..

[5]  Mitsuji Monta,et al.  Harvesting robot for strawberry grown on annual hill top (part 1) manufacture of the first prototype robot and fundamental harvesting experiment. , 2000 .

[6]  José Blasco,et al.  A new method for pedicel/peduncle detection and size assessment of grapevine berries and other fruits by image analysis , 2014 .

[7]  Jake K. Aggarwal,et al.  Stochastic Analysis of Stereo Quantization Error , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Bülent Sankur,et al.  Survey over image thresholding techniques and quantitative performance evaluation , 2004, J. Electronic Imaging.

[9]  Jun Lu,et al.  Virtual manipulator-based binocular stereo vision positioning system and errors modelling , 2010, Machine Vision and Applications.

[10]  J. Bontsema,et al.  An Autonomous Robot for Harvesting Cucumbers in Greenhouses , 2002, Auton. Robots.

[11]  Wei Li,et al.  Design and Experiment of Intelligent Grape Bagging Robot , 2013 .

[12]  Kenta Shigematsu,et al.  Evaluation of a strawberry-harvesting robot in a field test , 2010 .

[13]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[14]  Ribana Roscher,et al.  Image based evaluation for the detection of cluster parameters in grapevine , 2015 .

[15]  Min Ye,et al.  Fault-Tolerant Design of a Limited Universal Fruit-Picking End-Effector Based on Vision-Positioning Error , 2016 .

[16]  Mitsuji Monta,et al.  Robotic Harvesting System for Eggplants Trained in V-Shape (Part 2)-Harvesting Experiment for Eggplants- , 2003 .

[17]  Xiangjun Zou,et al.  Localisation of litchi in an unstructured environment using binocular stereo vision , 2016 .

[18]  E. J. van Henten,et al.  Field Test of an Autonomous Cucumber Picking Robot , 2003 .

[19]  Xiaoyan Hu,et al.  A Quantitative Evaluation of Confidence Measures for Stereo Vision , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  E. J. van Henten,et al.  Stem localization of sweet-pepper plants using the support wire as a visual cue , 2014 .

[21]  José Luis Pons Rovira,et al.  Machine Vision and Applications Manuscript-nr. a Vision System Based on a Laser Range--nder Applied to Robotic Fruit Harvesting , 2022 .

[22]  Jiri Matas,et al.  Robust Detection of Lines Using the Progressive Probabilistic Hough Transform , 2000, Comput. Vis. Image Underst..

[23]  Yibin Ying,et al.  Recognition of clustered tomatoes based on binocular stereo vision , 2014 .

[24]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Juan Feng,et al.  Location of apples in trees using stereoscopic vision , 2015, Comput. Electron. Agric..

[26]  Dani Martínez,et al.  Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination , 2014 .

[27]  E. J. van Henten,et al.  An Autonomous Robot for De-leafing Cucumber Plants grown in a High-wire Cultivation System , 2006 .

[28]  Kuan Chong Ting,et al.  Harvesting End-effector for Inverted Single Truss Tomato Production Systems , 1998 .

[29]  David Hernández-López,et al.  Vineyard yield estimation by automatic 3D bunch modelling in field conditions , 2015, Comput. Electron. Agric..

[30]  Mitsuji Monta,et al.  Basic Studies on Robot to work in Vineyard (Part 2) , 1994 .

[31]  Siddhartha S. Mehta,et al.  Vision-based control of robotic manipulator for citrus harvesting , 2014 .

[32]  Jiri Matas,et al.  Gradient based progressive probabilistic Hough transform , 2001 .

[33]  Wang Feng,et al.  Obstacle identification of citrus harvesting robot. , 2009 .