Green Grape Detection and Picking-Point Calculation in a Night-Time Natural Environment Using a Charge-Coupled Device (CCD) Vision Sensor with Artificial Illumination

Night-time fruit-picking technology is important to picking robots. This paper proposes a method of night-time detection and picking-point positioning for green grape-picking robots to solve the difficult problem of green grape detection and picking in night-time conditions with artificial lighting systems. Taking a representative green grape named Centennial Seedless as the research object, daytime and night-time grape images were captured by a custom-designed visual system. Detection was conducted employing the following steps: (1) The RGB (red, green and blue). Color model was determined for night-time green grape detection through analysis of color features of grape images under daytime natural light and night-time artificial lighting. The R component of the RGB color model was rotated and the image resolution was compressed; (2) The improved Chan–Vese (C–V) level set model and morphological processing method were used to remove the background of the image, leaving out the grape fruit; (3) Based on the character of grape vertical suspension, combining the principle of the minimum circumscribed rectangle of fruit and the Hough straight line detection method, straight-line fitting for the fruit stem was conducted and the picking point was calculated using the stem with an angle of fitting line and vertical line less than 15°. The visual detection experiment results showed that the accuracy of grape fruit detection was 91.67% and the average running time of the proposed algorithm was 0.46 s. The picking-point calculation experiment results showed that the highest accuracy for the picking-point calculation was 92.5%, while the lowest was 80%. The results demonstrate that the proposed method of night-time green grape detection and picking-point calculation can provide technical support to the grape-picking robots.

[1]  Masayuki Inaba,et al.  Reasoning-based vision recognition for agricultural humanoid robot toward tomato harvesting , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Chau Ton,et al.  Multiple camera fruit localization using a particle filter , 2017, Comput. Electron. Agric..

[3]  Satoshi Yamamoto,et al.  Development of a Stationary Robotic Strawberry Harvester with a Picking Mechanism that Approaches the Target Fruit from Below , 2014 .

[4]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[5]  Salviano F. S. P. Soares,et al.  Automatic detection of bunches of grapes in natural environment from color images , 2012, J. Appl. Log..

[6]  C. Kammann,et al.  Biochar and biochar-compost as soil amendments to a vineyard soil: Influences on plant growth, nutrient uptake, plant health and grape quality , 2014 .

[7]  Petr Dolezel,et al.  Detection of grapes in natural environment using feedforward neural network as a classifier , 2016, 2016 SAI Computing Conference (SAI).

[8]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[9]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[10]  James Patrick Underwood,et al.  Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry , 2016, Sensors.

[11]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[12]  Dennis Jarvis,et al.  Estimating mango crop yield using image analysis using fruit at 'stone hardening' stage and night time imaging , 2014 .

[13]  J. Bontsema,et al.  An Autonomous Robot for Harvesting Cucumbers in Greenhouses , 2002, Auton. Robots.

[14]  Sanjiv Singh,et al.  Yield estimation in vineyards by visual grape detection , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Matthew N. Dailey,et al.  Texture-based fruit detection , 2014, Precision Agriculture.

[16]  Miao Liang,et al.  A hybrid H component histogram threshold and sparse field level set algorithm for litchi image automatic segmentation , 2011, 2011 International Conference on Electric Information and Control Engineering.

[17]  S. Kitamura,et al.  Improvement of the Ability to Recognize Sweet Peppers for Picking Robot in Greenhouse Horticulture , 2006, 2006 SICE-ICASE International Joint Conference.

[18]  Raphael Linker,et al.  Apple detection in nighttime tree images using the geometry of light patches around highlights , 2015, Comput. Electron. Agric..

[19]  Rainer Goebel,et al.  From Coarse to Fine? Spatial and Temporal Dynamics of Cortical Face Processing , 2010, Cerebral cortex.

[20]  Qin Zhang,et al.  Detection of cherry tree branches with full foliage in planar architecture for automated sweet-cherry harvesting , 2016 .

[21]  C. Glasbey,et al.  Automatic fruit recognition and counting from multiple images , 2014 .

[22]  Scarlett Liu,et al.  Automatic grape bunch detection in vineyards with an SVM classifier , 2015, J. Appl. Log..

[23]  Bart De Dobbelaer,et al.  Semi-automatic level set segmentation of liver tumors combining a spiral-scanning technique with supervised fuzzy pixel classification , 2010, Medical Image Anal..

[24]  Gaston Lefranc,et al.  Olive Fruits Recognition Using Neural Networks , 2013, ITQM.

[25]  Zhao Dean,et al.  System Design and Control of an Apple Harvesting Robot , 2020, ArXiv.

[26]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[27]  Ying Zhang,et al.  Automatic recognition vision system guided for apple harvesting robot , 2012, Comput. Electr. Eng..

[28]  Lei Zhang,et al.  Active contours with selective local or global segmentation: A new formulation and level set method , 2010, Image Vis. Comput..

[29]  E. J. van Henten,et al.  Stem localization of sweet-pepper plants using the support wire as a visual cue , 2014 .

[30]  Kaveh Mollazade,et al.  Recognition and localization of ripen tomato based on machine vision , 2011 .

[31]  Tony F. Chan,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[32]  A R Smith,et al.  Color Gamut Transformation Pairs , 1978 .

[33]  Grzegorz Cielniak,et al.  3D-vision based detection, localization, and sizing of broccoli heads in the field , 2017, J. Field Robotics.

[34]  Takashi Kataoka,et al.  Fruit detection system and an end effector for robotic harvesting of Fuji apples , 2010 .

[35]  W. S. Qureshi,et al.  Machine vision for counting fruit on mango tree canopies , 2017, Precision Agriculture.

[36]  Dani Martínez,et al.  A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm , 2014, Sensors.