Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review

The utilization of machine vision and its associated algorithms improves the efficiency, functionality, intelligence, and remote interactivity of harvesting robots in complex agricultural environments. Machine vision and its associated emerging technology promise huge potential in advanced agricultural applications. However, machine vision and its precise positioning still have many technical difficulties, making it difficult for most harvesting robots to achieve true commercial applications. This article reports the application and research progress of harvesting robots and vision technology in fruit picking. The potential applications of vision and quantitative methods of localization, target recognition, 3D reconstruction, and fault tolerance of complex agricultural environment are focused, and fault-tolerant technology designed for utilization with machine vision and robotic systems are also explored. The two main methods used in fruit recognition and localization are reviewed, including digital image processing technology and deep learning-based algorithms. The future challenges brought about by recognition and localization success rates are identified: target recognition in the presence of illumination changes and occlusion environments; target tracking in dynamic interference-laden environments, 3D target reconstruction, and fault tolerance of the vision system for agricultural robots. In the end, several open research problems specific to recognition and localization applications for fruit harvesting robots are mentioned, and the latest development and future development trends of machine vision are described.

[1]  Yuanshen Zhao,et al.  Robust Tomato Recognition for Robotic Harvesting Using Feature Images Fusion , 2016, Sensors.

[2]  Zhen Liu,et al.  Visual positioning technology of picking robots for dynamic litchi clusters with disturbance , 2018, Comput. Electron. Agric..

[3]  Wang Ning,et al.  Research development of fruit and vegetable harvesting robots in China. , 2010 .

[4]  et al. Tahir.  Extracting accurate time domain features from vibration signals for reliable classification of bearing faults , 2018 .

[5]  Jochen Hemming,et al.  Fruit Detectability Analysis for Different Camera Positions in Sweet-Pepper † , 2014, Sensors.

[6]  J.L. Pons,et al.  A machine vision system using a laser radar applied to robotic fruit harvesting , 1999, Proceedings IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications (CVBVS'99).

[7]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.

[8]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[9]  David C. Slaughter,et al.  Color Vision in Robotic Fruit Harvesting , 1987 .

[10]  Q. Zhang,et al.  Apple crop-load estimation with over-the-row machine vision system , 2016, Comput. Electron. Agric..

[11]  Yael Edan,et al.  Changing Task Objectives for Improved Sweet Pepper Detection for Robotic Harvesting , 2016, IEEE Robotics and Automation Letters.

[12]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[13]  Heikki Saari,et al.  Hyperspectral reflectance signatures and point clouds for precision agriculture by light weight UAV imaging system , 2012 .

[14]  Giorgio Grasso,et al.  Localization of spherical fruits for robotic harvesting , 2001, Machine Vision and Applications.

[15]  W. S. Qureshi,et al.  Machine vision for counting fruit on mango tree canopies , 2017, Precision Agriculture.

[16]  Vijay Kumar,et al.  Counting Apples and Oranges With Deep Learning: A Data-Driven Approach , 2017, IEEE Robotics and Automation Letters.

[17]  Thomas F. Burks,et al.  A Machine Vision Algorithm Combining Adaptive Segmentation and Shape Analysis for Orange Fruit Detection , 2010 .

[18]  Y. Hashimoto,et al.  Pattern recognition of fruit shape based on the concept of chaos and neural networks , 2000 .

[19]  Antonio-Javier Garcia-Sanchez,et al.  Wireless sensor network deployment for integrating video-surveillance and data-monitoring in precision agriculture over distributed crops , 2011 .

[20]  Marcello Chiaberge,et al.  Comparison of Satellite and UAV-Based Multispectral Imagery for Vineyard Variability Assessment , 2019, Remote. Sens..

[21]  Filiberto Pla,et al.  Feature extraction of spherical objects in image analysis: an application to robotic citrus harvesting , 1993 .

[22]  Shaocheng Tong,et al.  Adaptive Neural Networks Decentralized FTC Design for Nonstrict-Feedback Nonlinear Interconnected Large-Scale Systems Against Actuator Faults , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[23]  刘刚 Liu Gang,et al.  Segmentation Algorithm for Apple Recognition using Image Features and Artificial Neural Network , 2008 .

[24]  Shigehiko HAYASHI,et al.  Robotic Harvesting Technology for Fruit Vegetables in Protected Horticultural Production , 2005 .

[25]  Reza Ehsani,et al.  Identifying immature and mature pomelo fruits in trees by elliptical model fitting in the Cr–Cb color space , 2018, Precision Agriculture.

[26]  Grzegorz Cielniak,et al.  3D-vision based detection, localization, and sizing of broccoli heads in the field , 2017, J. Field Robotics.

[27]  Peeyush Soni,et al.  Development of an automatic grading machine for oil palm fresh fruits bunches (FFBs) based on machine vision , 2013 .

[28]  Ziyang Wang.  Robot Obstacle Avoidance and Navigation Control Algorithm Research Based on Multi-Sensor Information Fusion , 2018, 2018 11th International Conference on Intelligent Computation Technology and Automation (ICICTA).

[29]  Shingo Mabu,et al.  A Visual System of Citrus Picking Robot Using Convolutional Neural Networks , 2018, 2018 5th International Conference on Systems and Informatics (ICSAI).

[30]  Kushtrim Bresilla,et al.  Single-Shot Convolution Neural Networks for Real-Time Fruit Detection Within the Tree , 2019, Front. Plant Sci..

[31]  Jun Miura,et al.  3D Semantic Mapping in Greenhouses for Agricultural Mobile Robots with Robust Object Recognition Using Robots' Trajectory , 2018, 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[32]  C. Glasbey,et al.  Automatic fruit recognition and counting from multiple images , 2014 .

[33]  D. Bulanon,et al.  A Segmentation Algorithm for the Automatic Recognition of Fuji Apples at Harvest , 2002 .

[34]  Liu Gang,et al.  A segmentation algorithm for apple fruit recognition using artificial neural network. , 2008 .

[35]  Yael Edan,et al.  Robotic melon harvesting , 2000, IEEE Trans. Robotics Autom..

[36]  Mitsuji Monta,et al.  A Machine Vision System for Tomato Cluster Harvesting Robot , 2009 .

[37]  Mihai Oltean,et al.  Fruit recognition from images using deep learning , 2017, Acta Universitatis Sapientiae, Informatica.

[38]  E. J. van Henten,et al.  Optimal manipulator design for a cucumber harvesting robot , 2009 .

[39]  Yael Edan,et al.  Harvesting Robots for High‐value Crops: State‐of‐the‐art Review and Challenges Ahead , 2014, J. Field Robotics.

[40]  Yoav Sarig,et al.  Robotics of Fruit Harvesting: A State-of-the-art Review , 1993 .

[41]  Bin Jiang,et al.  Adaptive Fault-tolerant Neural Control for Large-scale Systems with Actuator Faults , 2019 .

[42]  Fumiya Iida,et al.  A field‐tested robotic harvesting system for iceberg lettuce , 2019, J. Field Robotics.

[43]  Bong Ki Lee,et al.  A Vision Servo System for Automated Harvest of Sweet Pepper in Korean Greenhouse Environment , 2019, Applied Sciences.

[44]  Naoshi Kondo,et al.  Robotics for Plant Production , 1998, Artificial Intelligence Review.

[45]  José Luis Pons Rovira,et al.  Machine Vision and Applications Manuscript-nr. a Vision System Based on a Laser Range--nder Applied to Robotic Fruit Harvesting , 2022 .

[46]  Won Suk Lee,et al.  Strawberry Yield Prediction Based on a Deep Neural Network Using High-Resolution Aerial Orthoimages , 2019, Remote. Sens..

[47]  Qin Zhang,et al.  Apple Tree Trunk and Branch Segmentation for Automatic Trellis Training Using Convolutional Neural Network Based Semantic Segmentation , 2018 .

[48]  Mark Yim,et al.  An integrated system for perception-driven autonomy with modular robots , 2017, Science Robotics.

[49]  Mazda Moattari,et al.  Fault-Tolerant Control of Teleoperation Systems with Flexible-Link Slave Robot and Disturbance Compensation , 2020 .

[50]  Qiang Lu,et al.  Obstacle Recognition Using Multi-Spectral Imaging for Citrus Picking Robot , 2011, 2011 Third Pacific-Asia Conference on Circuits, Communications and System (PACCS).

[51]  Liu Zhaoxiang,et al.  Apple maturity discrimination and positioning system in an apple harvesting robot , 2007 .

[52]  Xiangjun Zou,et al.  Localisation of litchi in an unstructured environment using binocular stereo vision , 2016 .

[53]  Vijay Kumar,et al.  The grand challenges of Science Robotics , 2018, Science Robotics.

[54]  E. J. van Henten,et al.  Field Test of an Autonomous Cucumber Picking Robot , 2003 .

[55]  Naoshi Kondo,et al.  Methods of Detecting Fruit by Visual Sensor Attached to Manipulator , 1985 .

[56]  Michael Mangan,et al.  L*a*b*Fruits: A Rapid and Robust Outdoor Fruit Detection System Combining Bio-Inspired Features with One-Stage Deep Learning Networks , 2020, Sensors.

[57]  Thomas F. Burks,et al.  A Real-time Machine Vision Algorithm for Robotic Citrus Harvesting , 2007 .

[58]  Jun Miura,et al.  Development of a mobile robot for harvest support in greenhouse horticulture — Person following and mapping , 2017, 2017 IEEE/SICE International Symposium on System Integration (SII).

[59]  Joseph Wang,et al.  Micro/nanorobots for biomedicine: Delivery, surgery, sensing, and detoxification , 2017, Science Robotics.

[60]  Changki Mo,et al.  Design, integration, and field evaluation of a robotic apple harvester , 2017, J. Field Robotics.

[61]  Kurt C. VerCauteren,et al.  Use of unmanned aircraft systems (UAS) and multispectral imagery for quantifying agricultural areas damaged by wild pigs , 2019, Crop Protection.

[62]  Rory C. Flemmer,et al.  Development of an autonomous kiwifruit picking robot , 2000, 2009 4th International Conference on Autonomous Robots and Agents.

[63]  Mogens Blanke,et al.  Original paper: Stereo vision with texture learning for fault-tolerant automatic baling , 2011 .

[64]  Seishi Ninomiya,et al.  On Plant Detection of Intact Tomato Fruits Using Image Analysis and Machine Learning Methods , 2014, Sensors.

[65]  Gaston Lefranc,et al.  Olive Fruits Recognition Using Neural Networks , 2013, ITQM.

[66]  Yael Edan,et al.  Development of a sweet pepper harvesting robot , 2020, J. Field Robotics.

[67]  Rahul Sukthankar,et al.  Classification of plant structures from uncalibrated image sequences , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[68]  Yuanshen Zhao,et al.  A review of key techniques of vision-based control for harvesting robot , 2016, Comput. Electron. Agric..

[69]  Jun Lu,et al.  Virtual manipulator-based binocular stereo vision positioning system and errors modelling , 2010, Machine Vision and Applications.

[70]  Qin Zhang,et al.  The Use of Agricultural Robots in Orchard Management , 2019, Robotics and automation for improving agriculture.

[71]  Yiliang Zeng,et al.  Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot , 2014 .

[72]  Ming Li,et al.  Review of research on agricultural vehicle autonomous guidance , 2009 .

[73]  Salviano F. S. P. Soares,et al.  Automatic detection of bunches of grapes in natural environment from color images , 2012, J. Appl. Log..

[74]  Tony E Grift,et al.  Variable field-of-view machine vision based row guidance of an agricultural robot , 2012 .

[75]  Diego Patino,et al.  Multispectral mapping in agriculture: Terrain mosaic using an autonomous quadcopter UAV , 2016, 2016 International Conference on Unmanned Aircraft Systems (ICUAS).

[76]  Wengang Zheng,et al.  New strawberry harvesting robot for elevated-trough culture , 2012 .

[77]  Seema Verma,et al.  Analysis and Detection of Fruit Defect Using Neural Network , 2017 .

[78]  Joris IJsselmuiden,et al.  Robot navigation in orchards with localization based on Particle filter and Kalman filter , 2019, Comput. Electron. Agric..

[79]  Xiangjun Zou,et al.  A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot , 2017 .

[80]  S. Hata,et al.  Development of a real-time machine vision system for the apple harvesting robot , 2004, SICE 2004 Annual Conference.

[81]  Tateshi Fujiura,et al.  Cherry-harvesting robot , 2008 .

[82]  Chao Chen,et al.  Fruit Detection and Segmentation for Apple Harvesting Using Visual Sensor in Orchards , 2019, Sensors.

[83]  Pengfei Li,et al.  Fruit recognition based on convolution neural network , 2016, 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD).

[84]  Takanori Fukao,et al.  An automated fruit harvesting robot by using deep learning , 2019, ROBOMECH Journal.

[85]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[86]  Shin-Dug Kim,et al.  A Code Based Fruit Recognition Method Via Image Convertion Using Multiple Features , 2014, 2014 International Conference on IT Convergence and Security (ICITCS).

[87]  Zhen Liu,et al.  The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment , 2018 .

[88]  Ho Seok Ahn,et al.  Improvements to and large‐scale evaluation of a robotic kiwifruit harvester , 2019, J. Field Robotics.

[89]  van E.J. Henten.  Greenhouse mechanization: State of the art and future perspective , 2006 .

[90]  Ping Zhang,et al.  Unsupervised Segmentation of Greenhouse Plant Images Based on Statistical Method , 2018, Scientific Reports.

[91]  Lars Grimstad,et al.  Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper , 2019, Comput. Electron. Agric..

[92]  Brijesh Verma,et al.  On-Tree Mango Fruit Size Estimation Using RGB-D Images , 2017, Sensors.

[93]  Jochen Hemming,et al.  Angle estimation between plant parts for grasp optimisation in harvest robots , 2019, Biosystems Engineering.

[94]  C. E. Schertz,et al.  Evaluating Shake Harvesting of Oranges for the Fresh Fruit Market , 1967 .

[95]  Jiang Bian,et al.  Design and implementation of an automatic peach-harvesting robot system , 2018, 2018 Tenth International Conference on Advanced Computational Intelligence (ICACI).

[96]  Jie Zhan,et al.  Comparison of two deep learning methods for ship target recognition with optical remotely sensed data , 2020, Neural Computing and Applications.

[97]  Zhe Zhang,et al.  End-effector with a bite mode for harvesting citrus fruit in random stalk orientation environment , 2019, Comput. Electron. Agric..

[98]  Yong He,et al.  Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications , 2018, Comput. Electron. Agric..

[99]  David C. Slaughter,et al.  Vision Guidance Of A Robotic Tree Fruit Harvester , 1985, Other Conferences.

[100]  Kaveh Mollazade,et al.  Recognition and localization of ripen tomato based on machine vision , 2011 .

[101]  Zhao Dean,et al.  System Design and Control of an Apple Harvesting Robot , 2020, ArXiv.

[102]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[103]  F. Albert,et al.  Optimised computer vision system for automatic pre-grading of citrus fruit in the field using a mobile platform , 2014, Precision Agriculture.

[104]  G Sandini,et al.  The AGROBOT project. , 1996, Advances in space research : the official journal of the Committee on Space Research.

[105]  Changki Mo,et al.  Effort towards robotic apple harvesting in Washington State , 2016 .

[106]  Tristan Perez,et al.  Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting—Combined Color and 3-D Information , 2017, IEEE Robotics and Automation Letters.

[107]  N. Kondo,et al.  A review of automation and robotics for the bio-industry , 2008 .

[108]  Swapnil Kulkarni,et al.  Automation for Agriculture , 2015 .

[109]  Ying Zhang,et al.  Research of Apple Harvesting Robot Based on Least Square Support Vector Machine , 2010, 2010 International Conference on Electrical and Control Engineering.

[110]  Yong He,et al.  Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios , 2019, Biosystems Engineering.

[111]  K. Folta,et al.  Breeding new varieties for controlled environments. , 2018, Plant biology.

[112]  Xiong Chen,et al.  A vision methodology for harvesting robot to detect cutting points on peduncles of double overlapping grape clusters in a vineyard , 2018, Comput. Ind..

[113]  Zhong,et al.  Segmentation methods of fruit image based on color difference , 2009 .

[114]  Xiangjun Zou,et al.  Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field , 2019, Sensors.

[115]  Feng Li,et al.  Intelligent active fault-tolerant system for multi-source integrated navigation system based on deep neural network , 2019, Neural Computing and Applications.

[116]  Jun Zhou,et al.  Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking , 2017, Comput. Electron. Agric..

[117]  Rui Li,et al.  Kiwifruit detection in field images using Faster R-CNN with VGG16 , 2019, IFAC-PapersOnLine.

[118]  Sudhanshu Sekhar Panda,et al.  Remote Sensing and Geospatial Technological Applications for Site-specific Management of Fruit and Nut Crops: A Review , 2010, Remote. Sens..

[119]  Ohad Ben-Shahar,et al.  Colour-agnostic shape-based 3D fruit detection for crop harvesting robots , 2016 .

[120]  Lvwen Huang,et al.  Ripe Fuji Apple Detection Model Analysis in Natural Tree Canopy , 2012 .

[121]  Kuan Chong Ting,et al.  Visual feedback guided robotic cherry tomato harvesting , 1996 .

[122]  Eric Fleury,et al.  Tracking Clinical Staff Behaviors in an Operating Room , 2019, Sensors.

[123]  Won Suk Lee,et al.  Green citrus detection using 'eigenfruit', color and circular Gabor texture features under natural outdoor conditions , 2011 .

[124]  Baylee Ogletree,et al.  FRUITS AND VEGETABLES , 2001 .

[125]  Jose L Pons,et al.  A SURVEY OF COMPUTER VISION METHODS FOR LOCATING FRUIT ON TREES , 2000 .

[126]  Zhenglin Wang,et al.  Mango Fruit Load Estimation Using a Video Based MangoYOLO—Kalman Filter—Hungarian Algorithm Method , 2019, Sensors.

[127]  Xiangjun Zou,et al.  Binocular vision measurement and its application in full-field convex deformation of concrete-filled steel tubular columns , 2018, Measurement.

[128]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[129]  Shigehiko Hayashi,et al.  Robotic Harvesting System for Eggplants , 2002 .

[130]  Jose L Pons,et al.  Design and implementation of an aided fruit‐harvesting robot (Agribot) , 1998 .

[131]  Lorenzo Comba,et al.  Unsupervised detection of vineyards by 3D point-cloud UAV photogrammetry for precision agriculture , 2018, Comput. Electron. Agric..

[132]  Jun Zhou,et al.  Shadow detection and removal in apple image segmentation under natural light conditions using an ultrametric contour map , 2019, Biosystems Engineering.

[133]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[134]  Simon X. Yang,et al.  Ripe tomato extraction for a harvesting robotic system , 2009, 2009 IEEE International Conference on Systems, Man and Cybernetics.

[135]  E. J. van Henten,et al.  Stem localization of sweet-pepper plants using the support wire as a visual cue , 2014 .

[136]  Sang-Heon Lee,et al.  Study on citrus fruit image using fisher linear discriminant analysis , 2011, 2011 IEEE International Conference on Computer Science and Automation Engineering.

[137]  S. Kitamura,et al.  Recognition and cutting system of sweet pepper for picking robot in greenhouse horticulture , 2005, IEEE International Conference Mechatronics and Automation, 2005.

[138]  Victor Alchanatis,et al.  A Multispectral Imaging Analysis for Enhancing Citrus Fruit Detection , 2010 .

[139]  Ben Somers,et al.  Multitemporal Chlorophyll Mapping in Pome Fruit Orchards from Remotely Piloted Aircraft Systems , 2019, Remote. Sens..

[140]  Liguo Zhang,et al.  Study on the Fruit Recognition System Based on Machine Vision , 2016 .

[141]  G. Rabatel,et al.  Magali: a self-propelled robot to pick apples , 1987 .

[142]  Salah Sukkarieh,et al.  Orchard fruit segmentation using multi-spectral feature learning , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[143]  Ahmad Al-Mallahi,et al.  A novel image processing algorithm to separate linearly clustered kiwifruits , 2019, Biosystems Engineering.

[144]  P. L. Watts,et al.  Economic Considerations in Industrial Robotics , 1983 .

[145]  Avital Bechar,et al.  Robotics in horticultural field production. , 2010 .

[146]  Anhar Risnumawan,et al.  End-to-end coconut harvesting robot , 2016, 2016 International Electronics Symposium (IES).

[147]  Min Ye,et al.  Fault-Tolerant Design of a Limited Universal Fruit-Picking End-Effector Based on Vision-Positioning Error , 2016 .

[148]  Jun Zhao,et al.  On-tree fruit recognition using texture properties and color data , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[149]  Yael Edan,et al.  Controlled Lighting and Illumination-Independent Target Detection for Real-Time Cost-Efficient Applications. The Case Study of Sweet Pepper Robotic Harvesting , 2019, Sensors.

[150]  William MacKunis,et al.  Nonlinear Robust Visual Servo Control for Robotic Citrus Harvesting , 2014 .

[151]  Juan Feng,et al.  Location of apples in trees using stereoscopic vision , 2015, Comput. Electron. Agric..

[152]  Won Suk Lee,et al.  Correction to: Detection and counting of immature green citrus fruit based on the Local Binary Patterns (LBP) feature using illumination-normalized images , 2018, Precision Agriculture.

[153]  Xiangjun Zou,et al.  Vision-based extraction of spatial information in grape clusters for harvesting robots , 2016 .

[154]  Yibin Ying,et al.  Recognition of clustered tomatoes based on binocular stereo vision , 2014 .

[155]  J. C. Siemens,et al.  Mechanics of Soil as Influenced by Model Tillage Tools , 1965 .