Fruit Detection and Recognition Based on Deep Learning for Automatic Harvesting: An Overview and Review

Continuing progress in machine learning (ML) has led to significant advancements in agricultural tasks. Due to its strong ability to extract high-dimensional features from fruit images, deep learning (DL) is widely used in fruit detection and automatic harvesting. Convolutional neural networks (CNN) in particular have demonstrated the ability to attain accuracy and speed levels comparable to those of humans in some fruit detection and automatic harvesting fields. This paper presents a comprehensive overview and review of fruit detection and recognition based on DL for automatic harvesting from 2018 up to now. We focus on the current challenges affecting fruit detection performance for automatic harvesting: the scarcity of high-quality fruit datasets, fruit detection of small targets, fruit detection in occluded and dense scenarios, fruit detection of multiple scales and multiple species, and lightweight fruit detection models. In response to these challenges, we propose feasible solutions and prospective future development trends. Future research should prioritize addressing these current challenges and improving the accuracy, speed, robustness, and generalization of fruit vision detection systems, while reducing the overall complexity and cost. This paper hopes to provide a reference for follow-up research in the field of fruit detection and recognition based on DL for automatic harvesting.

[1]  Xiaomeng Lv,et al.  Object Detection and Recognition Techniques Based on Digital Image Processing and Traditional Machine Learning for Fruit and Vegetable Harvesting Robots: An Overview and Review , 2023, Agronomy.

[2]  Hui Liu,et al.  A Multiscale Lightweight and Efficient Model Based on YOLOv7: Applied to Citrus Orchard , 2022, Plants.

[3]  Chunyu Deng,et al.  Efficient tomato harvesting robot based on image processing and deep learning , 2022, Precision Agriculture.

[4]  J. Ning,et al.  Visual recognition of cherry tomatoes in plant factory based on improved deep instance segmentation , 2022, Comput. Electron. Agric..

[5]  Jun Yu Li,et al.  Partition harvesting of a column-comb litchi harvester based on 3D clustering , 2022, Computers and Electronics in Agriculture.

[6]  Qingchun Feng,et al.  Advance of Target Visual Information Acquisition Technology for Fresh Fruit Robotic Harvesting: A Review , 2022, Agronomy.

[7]  Liangliang Niu,et al.  Recognition of fruits and vegetables with similar‐color background in natural environment: A survey , 2022, J. Field Robotics.

[8]  Xianju Yuan,et al.  A review on structural development and recognition–localization methods for end-effector of fruit–vegetable picking robots , 2022, International Journal of Advanced Robotic Systems.

[9]  M. K. Hazarika,et al.  Maturity detection of tomatoes using Transfer Learning , 2022, Measurement: Food.

[10]  A. Tiwari,et al.  Pre-trained deep learning-based classification of jujube fruits according to their maturity level , 2022, Neural Computing and Applications.

[11]  Yan Zhang,et al.  Detection and Segmentation of Mature Green Tomatoes Based on Mask R-CNN with Automatic Image Acquisition Approach , 2021, Sensors.

[12]  Dongjian He,et al.  Tomato Young Fruits Detection Method under Near Color Background Based on Improved Faster R-CNN with Attention Mechanism , 2021, Agriculture.

[13]  Hossein Khosravi,et al.  Real-time recognition of on-branch olive ripening stages by a deep convolutional neural network , 2021 .

[14]  O. E. Apolo-Apolo,et al.  Intelligent Fruit Yield Estimation for Orchards Using Deep Learning Based Semantic Segmentation Techniques—A Review , 2021, Frontiers in Plant Science.

[15]  Zhongzhi Han,et al.  E-AlexNet: quality evaluation of strawberry based on machine learning , 2021, Journal of Food Measurement and Characterization.

[16]  Johan Potgieter,et al.  Automation in Agriculture by Machine and Deep Learning Techniques: A Review of Recent Developments , 2021, Precision Agriculture.

[17]  Fuzeng Yang,et al.  A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5 , 2021, Remote. Sens..

[18]  Abdulkader Helwan,et al.  Deep Learning Based on Residual Networks for Automatic Sorting of Bananas , 2021 .

[19]  Shajin Prince,et al.  Recognition of Bloom/Yield in Crop Images Using Deep Learning Models for Smart Agriculture: A Review , 2021, Agronomy.

[20]  Anuja Bhargava,et al.  Fruits and vegetables quality evaluation using computer vision: A review , 2021, J. King Saud Univ. Comput. Inf. Sci..

[21]  Subramanian Parvathi,et al.  Detection of maturity stages of coconuts in complex background using Faster R-CNN model , 2021 .

[22]  R. Salomón-Torres,et al.  Evaluation of Convolutional Neural Networks’ Hyperparameters with Transfer Learning to Determine Sorting of Ripe Medjool Dates , 2021, Agriculture.

[23]  Zhao Dean,et al.  System Design and Control of an Apple Harvesting Robot , 2020, ArXiv.

[24]  Rui Li,et al.  Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review , 2020, Comput. Electron. Agric..

[25]  Salah Sukkarieh,et al.  Design and evaluation of a modular robotic plum harvesting system utilizing soft components , 2020, J. Field Robotics.

[26]  Changying Li,et al.  Deep learning image segmentation and extraction of blueberry fruit traits associated with harvestability and yield , 2020, Horticulture Research.

[27]  Hongjun Wang,et al.  Detection of Fruit-Bearing Branches and Localization of Litchi Clusters for Vision-Based Harvesting Robots , 2020, IEEE Access.

[28]  A. Milella,et al.  Deep neural networks for grape bunch segmentation in natural images from a consumer-grade camera , 2020, Precision Agriculture.

[29]  Mukesh Tripathi,et al.  A role of computer vision in fruits and vegetables among various horticulture products of agriculture fields: A survey , 2020 .

[30]  Fernando Auat Cheein,et al.  Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation , 2020, Comput. Electron. Agric..

[31]  Zhenbo Li,et al.  Vegetable Recognition and Classification Based on Improved VGG Deep Learning Network Model , 2020, Int. J. Comput. Intell. Syst..

[32]  Zhengang Yang,et al.  Visual detection of green mangoes by an unmanned aerial vehicle in orchards based on a deep learning method , 2020, Biosystems Engineering.

[33]  Jing Zhang,et al.  Multi-class object detection using faster R-CNN and estimation of shaking locations for automated shake-and-catch apple harvesting , 2020, Comput. Electron. Agric..

[34]  Xiangjun Zou,et al.  Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review , 2020, Frontiers in Plant Science.

[35]  Marco Mora,et al.  A Review of Convolutional Neural Network Applied to Fruit Image Processing , 2020, Applied Sciences.

[36]  Yuyu Tian,et al.  Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot , 2020, Comput. Electron. Agric..

[37]  Weikuan Jia,et al.  Apple harvesting robot under information technology: A review , 2020 .

[38]  Mohammad Momeny,et al.  Classification of sour lemons based on apparent defects using stochastic pooling mechanism in deep convolutional neural networks , 2020 .

[39]  Jing Zhang,et al.  Deep learning based segmentation for automated training of apple trees on trellis wires , 2020, Comput. Electron. Agric..

[40]  Dongjian He,et al.  Deep Learning Approach for Apple Edge Detection to Remotely Monitor Apple Growth in Orchards , 2020, IEEE Access.

[41]  Shaohua Wan,et al.  Faster R-CNN for multi-class fruit detection using a robotic vision system , 2020, Comput. Networks.

[42]  Yael Edan,et al.  Development of a sweet pepper harvesting robot , 2020, J. Field Robotics.

[43]  Yueju Xue,et al.  Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images , 2020, Precision Agriculture.

[44]  Chao Chen,et al.  Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards , 2019, Comput. Electron. Agric..

[45]  Johan Potgieter,et al.  Plant Disease Detection and Classification by Deep Learning , 2019, Plants.

[46]  Chao Chen,et al.  Fruit Detection and Segmentation for Apple Harvesting Using Visual Sensor in Orchards , 2019, Sensors.

[47]  Ghulam Muhammad,et al.  Date Fruit Classification for Robotic Harvesting in a Natural Environment Using Deep Learning , 2019, IEEE Access.

[48]  Lars Grimstad,et al.  An autonomous strawberry‐harvesting robot: Design, development, integration, and field evaluation , 2019, J. Field Robotics.

[49]  Xu Yan,et al.  An Improved Faster R-CNN for Small Object Detection , 2019, IEEE Access.

[50]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[51]  Fumiya Iida,et al.  A field‐tested robotic harvesting system for iceberg lettuce , 2019, J. Field Robotics.

[52]  Jochen Hemming,et al.  Angle estimation between plant parts for grasp optimisation in harvest robots , 2019, Biosystems Engineering.

[53]  Chris McCarthy,et al.  Deep learning - Method overview and review of use for fruit detection and yield estimation , 2019, Comput. Electron. Agric..

[54]  Aalap Doshi,et al.  A comprehensive review on automation in agriculture using artificial intelligence , 2019, Artificial Intelligence in Agriculture.

[55]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.

[56]  S. Nof,et al.  Early Detection of Tomato Spotted Wilt Virus by Hyperspectral Imaging and Outlier Removal Auxiliary Classifier Generative Adversarial Nets (OR-AC-GAN) , 2019, Scientific Reports.

[57]  Wen Zhang,et al.  A review on weed detection using ground-based machine vision and image processing techniques , 2019, Comput. Electron. Agric..

[58]  Lars Grimstad,et al.  Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper , 2019, Comput. Electron. Agric..

[59]  Xiangjun Zou,et al.  Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field , 2019, Sensors.

[60]  Rafael Rieder,et al.  Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review , 2018, Comput. Electron. Agric..

[61]  Jayme G. A. Barbedo,et al.  Impact of dataset size and variety on the effectiveness of deep learning and transfer learning for plant disease classification , 2018, Comput. Electron. Agric..

[62]  Mike Thelwall,et al.  Google Scholar, Web of Science, and Scopus: a systematic comparison of citations in 252 subject categories , 2018, J. Informetrics.

[63]  Chen Li,et al.  High performance vegetable classification from images based on AlexNet deep learning model , 2018 .

[64]  Xu Liu,et al.  Deep learning for smart agriculture: Concepts, tools, applications, and opportunities , 2018 .

[65]  Zhen Liu,et al.  Visual positioning technology of picking robots for dynamic litchi clusters with disturbance , 2018, Comput. Electron. Agric..

[66]  Wei Zou,et al.  Design and test of robotic harvesting system for cherry tomato , 2018 .

[67]  Armin B. Cremers,et al.  DeepCotton: in-field cotton segmentation using deep fully convolutional network , 2017, J. Electronic Imaging.

[68]  Giulio Reina,et al.  A Survey of Ranging and Imaging Techniques for Precision Agriculture Phenotyping , 2017, IEEE/ASME Transactions on Mechatronics.

[69]  S. Wolfert,et al.  Big Data in Smart Farming – A review , 2017 .

[70]  Qin Zhang,et al.  Effect of Fruit Location on Apple Detachment with Mechanical Shaking , 2017 .

[71]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[72]  Tristan Perez,et al.  Autonomous Sweet Pepper Harvesting for Protected Cropping Systems , 2017, IEEE Robotics and Automation Letters.

[73]  Vijay Kumar,et al.  Counting Apples and Oranges With Deep Learning: A Data-Driven Approach , 2017, IEEE Robotics and Automation Letters.

[74]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[75]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  E. J. van Henten,et al.  Stem localization of sweet-pepper plants using the support wire as a visual cue , 2014 .

[77]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[78]  Ying Zhang,et al.  Automatic recognition vision system guided for apple harvesting robot , 2012, Comput. Electr. Eng..

[79]  V. Alchanatis,et al.  Low and high-level visual feature-based apple detection from multi-modal images , 2010, Precision Agriculture.

[80]  Kenta Shigematsu,et al.  Evaluation of a strawberry-harvesting robot in a field test , 2010 .

[81]  W. S. Lee,et al.  Green citrus detection using hyperspectral imaging , 2009 .

[82]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[83]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[84]  Germán H. Alférez,et al.  Image recognition of Legacy blueberries in a Chilean smart farm through deep learning , 2020, Comput. Electron. Agric..

[85]  Juntao Xiong,et al.  Semantic Segmentation of Litchi Branches Using DeepLabV3+ Model , 2020, IEEE Access.

[86]  Rui Li,et al.  Kiwifruit detection in field images using Faster R-CNN with VGG16 , 2019, IFAC-PapersOnLine.

[87]  Young K. Chang,et al.  Current and future applications of statistical machine learning algorithms for agricultural machine vision systems , 2019, Comput. Electron. Agric..

[88]  Qin Zhang,et al.  Apple Tree Trunk and Branch Segmentation for Automatic Trellis Training Using Convolutional Neural Network Based Semantic Segmentation , 2018 .

[89]  Raja Purushothaman,et al.  Tomato crop disease classification using pre-trained deep learning algorithm , 2018 .

[90]  Mitsuji Monta,et al.  Development of an End-Effector for a Tomato Cluster Harvesting Robot , 2010 .

[91]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.