A review of key techniques of vision-based control for harvesting robot

We presented a review on the representative vision schemes for harvesting robots.We reviewed hand-eye coordination techniques and their applications in harvesting robots.We presented some fruit or vegetable harvesting robots and their vision control techniques.We described and discussed the challenges and feature trends for robotic harvesting. Although there is a rapid development of agricultural robotic technologies, a lack of access to robust fruit recognition and precision picking capabilities has limited the commercial application of harvesting robots. On the other hand, recent advances in key techniques in vision-based control have improved this situation. These techniques include vision information acquisition strategies, fruit recognition algorithms, and eye-hand coordination methods. In a fruit or vegetable harvesting robot, vision control is employed to solve two major problems in detecting objects in tree canopies and picking objects using visual information. This paper presents a review on these key vision control techniques and their potential applications in fruit or vegetable harvesting robots. The challenges and feature trends of applying these vision control techniques in harvesting robots are also described and discussed in the review.

[1]  Dean Zhao,et al.  Research on the Strategy of Advancing Harvest Efficiency of Fruit Harvest Robot in the Oscillation Conditions , 2011, 2011 Third International Conference on Intelligent Human-Machine Systems and Cybernetics.

[2]  Yuanshen Zhao,et al.  Robust Tomato Recognition for Robotic Harvesting Using Feature Images Fusion , 2016, Sensors.

[3]  Mikio Umeda,et al.  Development of “STORK”, a watermelon-harvesting robot , 1999, Artificial Life and Robotics.

[4]  J. Bontsema,et al.  An Autonomous Robot for Harvesting Cucumbers in Greenhouses , 2002, Auton. Robots.

[5]  Matthew N. Dailey,et al.  Texture-based fruit detection , 2014, Precision Agriculture.

[6]  Yael Edan,et al.  Systems Engineering of Agricultural Robot Design , 1994, IEEE Trans. Syst. Man Cybern. Syst..

[7]  David C. Slaughter,et al.  Color Vision in Robotic Fruit Harvesting , 1987 .

[8]  T. F. Burks,et al.  Current Developments in Automated Citrus Harvesting , 2004 .

[9]  Yibin Ying,et al.  Detecting citrus in a tree canopy using infrared thermal imaging , 2004, SPIE Optics East.

[10]  Jizhan Liu,et al.  A Multi-Sensory End-Effector for Spherical Fruit Harvesting Robot , 2007, 2007 IEEE International Conference on Automation and Logistics.

[11]  Lvwen Huang,et al.  Ripe Fuji Apple Detection Model Analysis in Natural Tree Canopy , 2012 .

[12]  José Luis Pons Rovira,et al.  Machine Vision and Applications Manuscript-nr. a Vision System Based on a Laser Range--nder Applied to Robotic Fruit Harvesting , 2022 .

[13]  Tomoo Shiigi,et al.  Accurate Position Detecting during Asparagus Spear Harvesting using a Laser Sensor , 2013 .

[14]  Kuan Chong Ting,et al.  Visual feedback guided robotic cherry tomato harvesting , 1996 .

[15]  Ofer Levi,et al.  Detection of Green Apples in Hyperspectral Images of Apple-Tree Foliage Using Machine Vision , 2007 .

[16]  Victor Alchanatis,et al.  Image fusion of visible and thermal images for fruit detection. , 2009 .

[17]  Qin Zhang,et al.  A knot-tying end-effector for robotic hop twining , 2013 .

[18]  G Sandini,et al.  The AGROBOT project. , 1996, Advances in space research : the official journal of the Committee on Space Research.

[19]  E. J. van Henten,et al.  Collision-free inverse kinematics of the redundant seven-link manipulator used in a cucumber picking robot , 2010 .

[20]  R. Noble,et al.  AE—Automation and Emerging Technologies , 2001 .

[21]  G. Rabatel,et al.  Magali: a self-propelled robot to pick apples , 1987 .

[22]  Yoshiaki Shirai,et al.  Guiding a robot by visual feedback in assembling tasks , 1973, Pattern Recognit..

[23]  Liang Gong,et al.  Detecting tomatoes in greenhouse scenes by combining AdaBoost classifier and colour analysis , 2016 .

[24]  Philippe Martinet,et al.  Vision-Based Control in Driving Assistance of Agricultural Vehicles , 1998, Int. J. Robotics Res..

[25]  Vijay Subramanian,et al.  Engineering and Horticultural Aspects of Robotic Fruit Harvesting: Opportunities and Constraints , 2005 .

[26]  Ying Zhang,et al.  Automatic recognition vision system guided for apple harvesting robot , 2012, Comput. Electr. Eng..

[27]  Sang-Heon Lee,et al.  Review on fruit harvesting method for potential use of automatic fruit harvesting systems , 2011 .

[28]  Yibin Ying,et al.  Recognition of clustered tomatoes based on binocular stereo vision , 2014 .

[29]  D. Jayas,et al.  Applications of Thermal Imaging in Agriculture and Food Industry—A Review , 2011 .

[30]  William MacKunis,et al.  Nonlinear Robust Visual Servo Control for Robotic Citrus Harvesting , 2014 .

[31]  Heon Hwang,et al.  Strawberry Harvesting Robot for Bench-type Cultivation , 2012 .

[32]  L. Yang,et al.  A fruit recognition method for automatic harvesting , 2007, 2007 14th International Conference on Mechatronics and Machine Vision in Practice.

[33]  Yael Edan,et al.  Performance analysis of a human-robot collaborative target recognition system , 2012, Robotica.

[34]  Gregory D. Hager,et al.  Robot feedback control based on stereo vision: towards calibration-free hand-eye coordination , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[35]  Pan Qie-lu UNCALIBRATED 3D ROBOTIC VISUAL TRACKING BASED ON STEREO VISION , 2000 .

[36]  Satinder Kaur Brar,et al.  Optimisation of spray drying by response surface methodology for the production of Sinorhizobium meliloti powder formulation by using starch industry wastewater , 2013 .

[37]  Peng-Yung Woo,et al.  Calibration-free robotic eye-hand coordination based on an auto disturbance-rejection controller , 2004, IEEE Transactions on Robotics.

[38]  Juan Feng,et al.  Location of apples in trees using stereoscopic vision , 2015, Comput. Electron. Agric..

[39]  Yukitsugu Ishii,et al.  Development of a Harvesting End-Effector for Eggplants. , 2001 .

[40]  Zhao Dean,et al.  System Design and Control of an Apple Harvesting Robot , 2020, ArXiv.

[41]  Manjunath V. Joshi,et al.  Fruit Detection using Improved Multiple Features based Algorithm , 2011 .

[42]  Tateshi Fujiura,et al.  Cherry-harvesting robot , 2008 .

[43]  Noboru Noguchi,et al.  Development of a master-slave robot system for farm operations , 2004 .

[44]  Jun Lu,et al.  Virtual manipulator-based binocular stereo vision positioning system and errors modelling , 2010, Machine Vision and Applications.

[45]  F. E. Sistler,et al.  Robotics and intelligent machines in agriculture , 1987, IEEE J. Robotics Autom..

[46]  Rory C. Flemmer,et al.  Development of an autonomous kiwifruit picking robot , 2000, 2009 4th International Conference on Autonomous Robots and Agents.

[47]  Pedro M. B. Torres,et al.  Learning Approaches to Visual Control of Robotic Manipulators , 2010 .

[48]  E. J. van Henten,et al.  Collision-free Motion Planning for a Cucumber Picking Robot , 2003 .

[49]  Giuseppe Oriolo,et al.  Image-based visual servoing for nonholonomic mobile robots with central catadioptric camera , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[50]  Nong Sang,et al.  Detecting citrus fruits with highlight on tree based on fusion of multi-map , 2014 .

[51]  F Universit Recognition of kiwifruit in field based on Adaboost algorithm , 2013 .

[52]  Wang Hongjun,et al.  Recognition of ripe litchi in different illumination conditions based on Retinex image enhancement , 2013 .

[53]  Kaveh Mollazade,et al.  Recognition and localization of ripen tomato based on machine vision , 2011 .

[54]  R. C. Harrell,et al.  Robotic picking of citrus , 1990, Robotica.

[55]  V. Alchanatis,et al.  Low and high-level visual feature-based apple detection from multi-modal images , 2010, Precision Agriculture.

[56]  E. A. Parrish,et al.  Pictorial Pattern Recognition Applied to Fruit Harvesting , 1977 .

[57]  Yael Edan,et al.  An algorithm defining the motions of a citrus picking robot , 1990 .

[58]  David C. Slaughter,et al.  Discriminating Fruit for Robotic Harvest Using Color in Natural Outdoor Scenes , 1989 .

[59]  Giulio Reina,et al.  Agricultural robot for radicchio harvesting , 2006, J. Field Robotics.

[60]  Won Suk Lee,et al.  Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network , 2013, Precision Agriculture.

[61]  Gary A. Shaw,et al.  Hyperspectral Image Processing for Automatic Target Detection Applications , 2003 .

[62]  Siddhartha S. Mehta,et al.  Vision-based control of robotic manipulator for citrus harvesting , 2014 .

[63]  Baohua Zhang,et al.  Principles, developments and applications of computer vision for external quality inspection of fruits and vegetables: A review , 2014 .

[64]  Qin Zhang,et al.  A Review of Imaging Techniques for Plant Phenotyping , 2014, Sensors.

[65]  Thomas F. Burks,et al.  A Machine Vision Algorithm Combining Adaptive Segmentation and Shape Analysis for Orange Fruit Detection , 2010 .

[66]  EdanYael,et al.  Harvesting Robots for High-value Crops , 2014 .

[67]  Victor Alchanatis,et al.  Study on temporal variation in citrus canopy using thermal imaging for citrus fruit detection , 2008 .

[68]  C. Glasbey,et al.  Automatic fruit recognition and counting from multiple images , 2014 .

[69]  Yael Edan,et al.  Automation in Agriculture , 2009, Handbook of Automation.

[70]  D. Stajnko,et al.  Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging , 2004 .

[71]  Won Suk Lee,et al.  Green citrus detection using 'eigenfruit', color and circular Gabor texture features under natural outdoor conditions , 2011 .

[72]  Eric Claesen,et al.  Autonomous Fruit Picking Machine: A Robotic Apple Harvester , 2007, FSR.

[73]  Jose L Pons,et al.  A SURVEY OF COMPUTER VISION METHODS FOR LOCATING FRUIT ON TREES , 2000 .

[74]  T. Fujiura,et al.  3-D vision system of tomato production robot , 2003, Proceedings 2003 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2003).

[75]  G. E. Miles,et al.  Fruit location in a partially occluded image , 1984 .

[76]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[77]  Jun Zhao,et al.  On-tree fruit recognition using texture properties and color data , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[78]  Saeid Minaei,et al.  Robotic Harvesting of Rosa Damascena Using Stereoscopic Machine Vision , 2011 .

[79]  Shigehiko Hayashi,et al.  Robotic Harvesting System for Eggplants , 2002 .

[80]  Jose L Pons,et al.  Design and implementation of an aided fruit‐harvesting robot (Agribot) , 1998 .

[81]  Manuel A. Armada,et al.  Validation of a Multisensory System for Fruit Harvesting Robots in Lab Conditions , 2013, ROBOT.

[82]  W. S. Lee,et al.  Green citrus detection using hyperspectral imaging , 2009 .

[83]  Raphael Linker,et al.  Vision-based localisation of mature apples in tree images using convexity , 2014 .

[84]  Dani Martínez,et al.  A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm , 2014, Sensors.

[85]  Thomas Rath,et al.  Novel image processing approach for solving the overlapping problem in agriculture , 2013 .

[86]  Takeo Ojika,et al.  Cooperated operation of plural hand-robots for automatic harvest system , 1996 .

[87]  Simon X. Yang,et al.  Ripe tomato extraction for a harvesting robotic system , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[88]  Wei-Yun Yau,et al.  Robust hand-eye coordination , 1996, Adv. Robotics.

[89]  Arman Arefi,et al.  Development of an expert system based on wavelet transform and artificial neural networks for the ripe tomato harvesting robot , 2013 .

[90]  S. Kitamura,et al.  Recognition and cutting system of sweet pepper for picking robot in greenhouse horticulture , 2005, IEEE International Conference Mechatronics and Automation, 2005.

[91]  C. E. Schertz,et al.  Basic Considerations in Mechanizing Citrus Harvest , 1968 .

[92]  Michael J. Delwiche,et al.  Computer Vision to Locate Fruit on a Tree , 1985 .

[93]  Koichi Hashimoto,et al.  A review on vision-based control of robot manipulators , 2003, Adv. Robotics.

[94]  E. J. Pekkeriet,et al.  A robot for harvesting sweet-pepper in greenhouses , 2014 .

[95]  W. S. Lee,et al.  Multispectral Imaging for In-field Green Citrus Identification , 2007 .

[96]  Won Suk Lee,et al.  An Advanced Green Citrus Detection Algorithm Using Color Images and Neural Networks , 2011 .

[97]  Yael Edan,et al.  Robotic melon harvesting , 2000, IEEE Trans. Robotics Autom..

[98]  Yael Edan,et al.  Harvesting Robots for High‐value Crops: State‐of‐the‐art Review and Challenges Ahead , 2014, J. Field Robotics.

[99]  Takashi Kataoka,et al.  Fruit detection system and an end effector for robotic harvesting of Fuji apples , 2010 .

[100]  Kenta Shigematsu,et al.  Evaluation of a strawberry-harvesting robot in a field test , 2010 .

[101]  Colin Bradley,et al.  Adaptive neural network visual servo control for dynamic positioning of underwater vehicles , 2015, Neurocomputing.

[102]  Yiliang Zeng,et al.  Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot , 2014 .

[103]  Liang Gong,et al.  Computer vision recognition of stem and calyx in apples using near-infrared linear-array structured light and 3D reconstruction , 2015 .

[104]  Gregory D. Hager,et al.  Vision-based robot control , 1998 .

[105]  Sang-Heon Lee,et al.  Study on citrus fruit image data separability by segmentation methods , 2011 .

[106]  Moshe P. Mann,et al.  Harvest-order planning for a multiarm robotic harvester , 2014 .

[107]  Salviano F. S. P. Soares,et al.  Automatic detection of bunches of grapes in natural environment from color images , 2012, J. Appl. Log..

[108]  Wenhua Mao,et al.  Apple Location Method for the Apple Harvesting Robot , 2009, 2009 2nd International Congress on Image and Signal Processing.

[109]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[110]  Ting Yuan,et al.  Research on Key Technology of Truss Tomato Harvesting Robot in Greenhouse , 2013 .

[111]  Seishi Ninomiya,et al.  On Plant Detection of Intact Tomato Fruits Using Image Analysis and Machine Learning Methods , 2014, Sensors.

[112]  Gaston Lefranc,et al.  Olive Fruits Recognition Using Neural Networks , 2013, ITQM.

[113]  Koichi Osuka,et al.  Robust Control Systems of a Heavy Material Handling Agricultural Robot: A Case Study for Initial Cost Problem , 2007, IEEE Transactions on Control Systems Technology.

[114]  Guo Xiaoqing,et al.  An object detection method for quasi-circular fruits based on improved Hough transform. , 2010 .

[115]  Yael Edan,et al.  Human-robot collaboration for improved target recognition of agricultural robots , 2003, Ind. Robot.

[116]  S. Hata,et al.  Development of a real-time machine vision system for the apple harvesting robot , 2004, SICE 2004 Annual Conference.

[117]  Simon X. Yang,et al.  Ripe Tomato Recognition and Localization for a Tomato Harvesting Robotic System , 2009, 2009 International Conference of Soft Computing and Pattern Recognition.

[118]  E. J. van Henten,et al.  Field Test of an Autonomous Cucumber Picking Robot , 2003 .