Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review

Abstract Fruit detection and localization are essential for future agronomic management of fruit crops such as yield prediction, yield mapping and automated harvesting. However, to perform robust and efficient fruit detection and localization in orchard is a challenging task under variable illumination, low-resolutions and heavy occlusion by neighboring fruits, foliage, or branches. Therefore, researches of fruit detection and localization by getting more information of objects are essential. RGB-D (Red, Green, Blue -Depth) cameras are promising sensors and widely used in fruit detection and localization given that they provide depth information and infrared information in addition to RGB information. After presenting a discussion on the advantages and disadvantages of RGB-D cameras with different depth measurement principles and application fields, this paper reviews various types of RGB-D sensor systems and image processing methods used for fruit detection and localization in the field. Finally, major challenges for the successful application of RGB-D camera-based machine vision system, and potential future directions for the research and development in this area are discussed.

[1]  Rui Li,et al.  Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion , 2020, IEEE Access.

[2]  Ahmad Al-Mallahi,et al.  A novel image processing algorithm to separate linearly clustered kiwifruits , 2019, Biosystems Engineering.

[3]  Fernando Alfredo Auat Cheeín,et al.  Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry , 2017, Comput. Electron. Agric..

[4]  Siddhartha S. Mehta,et al.  Vision-based control of robotic manipulator for citrus harvesting , 2014 .

[5]  Kamel Barkaoui,et al.  Performability evaluation of server virtualized systems under bursty workload , 2018 .

[6]  Q. Zhang,et al.  Apple crop-load estimation with over-the-row machine vision system , 2016, Comput. Electron. Agric..

[7]  Josse De Baerdemaeker,et al.  Detection of red and bicoloured apples on tree with an RGB-D camera , 2016 .

[8]  Xin Zhang,et al.  Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting , 2020 .

[9]  Jordi Llorens,et al.  Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow , 2020, Comput. Electron. Agric..

[10]  Ke Xiao,et al.  An intelligent precision orchard pesticide spray technique based on the depth-of-field extraction algorithm , 2017, Comput. Electron. Agric..

[11]  Arman Arefi,et al.  Development of an expert system based on wavelet transform and artificial neural networks for the ripe tomato harvesting robot , 2013 .

[12]  Lee Sze Wei,et al.  Comparative Study of Intel R200, Kinect v2, and Primesense RGB-D Sensors Performance Outdoors , 2019, IEEE Sensors Journal.

[13]  José Dorado,et al.  Using depth cameras to extract structural parameters to assess the growth state and yield of cauliflower crops , 2016, Comput. Electron. Agric..

[14]  Brijesh Verma,et al.  On-Tree Mango Fruit Size Estimation Using RGB-D Images , 2017, Sensors.

[15]  Pål Johan From,et al.  Fruit Localization and Environment Perception for Strawberry Harvesting Robots , 2019, IEEE Access.

[16]  Huichuan Duan,et al.  Fast Recognition and Location of Target Fruit Based on Depth Information , 2019, IEEE Access.

[17]  Jing Zhang,et al.  Branch detection for apple trees trained in fruiting wall architecture using depth features and Regions-Convolutional Neural Network (R-CNN) , 2018, Comput. Electron. Agric..

[18]  Xiangjun Zou,et al.  In-field citrus detection and localisation based on RGB-D image analysis , 2019, Biosystems Engineering.

[19]  Manoj Karkee,et al.  Apple fruit size estimation using a 3D machine vision system , 2018, Information Processing in Agriculture.

[20]  Won Suk Lee,et al.  Green citrus detection using 'eigenfruit', color and circular Gabor texture features under natural outdoor conditions , 2011 .

[21]  Yueju Xue,et al.  Detection of passion fruits and maturity classification using Red-Green-Blue Depth images , 2018, Biosystems Engineering.

[22]  Xinxin Zhu,et al.  Experiments and Analysis of Close-Shot Identification of On-Branch Citrus Fruit with RealSense , 2018, Sensors.

[23]  Shigehiko Hayashi,et al.  Growth Measurement of a Community of Strawberries Using Three-Dimensional Sensor , 2015 .

[24]  Ludovico Minto,et al.  Operating Principles of Time-of-Flight Depth Cameras , 2016 .

[25]  Chao Chen,et al.  Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards , 2019, Comput. Electron. Agric..

[26]  Manoj Karkee,et al.  A HIERARCHICAL APPROACH TO APPLE IDENTIFICATION FOR ROBOTIC HARVESTING , 2016 .

[27]  Bin Wang,et al.  The Acquisition of Kiwifruit Feature Point Coordinates Based on the Spatial Coordinates of Image , 2015, CCTA.

[28]  Yael Edan,et al.  Development of a sweet pepper harvesting robot , 2020, J. Field Robotics.

[29]  David Reiser,et al.  3-D reconstruction of maize plants using a time-of-flight camera , 2018, Comput. Electron. Agric..

[30]  Yuanshen Zhao,et al.  A review of key techniques of vision-based control for harvesting robot , 2016, Comput. Electron. Agric..

[31]  Lutz Plümer,et al.  Low-Cost 3D Systems: Suitable Tools for Plant Phenotyping , 2014, Sensors.

[32]  Bernhard Höfle,et al.  Mobile low-cost 3D camera maize crop height measurements under field conditions , 2017, Precision Agriculture.

[33]  Kim L. Boyer,et al.  Color-Encoded Structured Light for Rapid Active Ranging , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Nelson L. Max,et al.  Structured Light-Based 3D Reconstruction System for Plants , 2015, Sensors.

[35]  Xiangjun Zou,et al.  Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review , 2020, Frontiers in Plant Science.

[36]  Guy Shani,et al.  Comparing RGB-D Sensors for Close Range Outdoor Agricultural Phenotyping , 2018, Sensors.

[37]  David C. Slaughter,et al.  Discriminating Fruit for Robotic Harvest Using Color in Natural Outdoor Scenes , 1989 .

[38]  Ilan Halachmi,et al.  3D Computer-vision system for automatically estimating heifer height and body mass , 2017, Biosystems Engineering.

[39]  Ohad Ben-Shahar,et al.  Colour-agnostic shape-based 3D fruit detection for crop harvesting robots , 2016 .

[40]  Ye Tian,et al.  SEGMENTATION OF APPLE POINT CLOUDS BASED ON ROI IN RGB IMAGES , 2019, INMATEH Agricultural Engineering.

[41]  Wang Yijie,et al.  A method to obtain the near-large fruit from apple image in orchard for single-arm apple harvesting robot , 2019, Scientia Horticulturae.

[42]  Dani Martínez,et al.  A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm , 2014, Sensors.

[43]  J. B. Li,et al.  Machine vision technology for detecting the external defects of fruits — a review , 2015 .

[44]  Chaoyong Shen,et al.  Estimating Tree Position, Diameter at Breast Height, and Tree Height in Real-Time Using a Mobile Phone with RGB-D SLAM , 2018, Remote. Sens..

[45]  Paulo Flores,et al.  Technology progress in mechanical harvest of fresh market apples , 2020, Comput. Electron. Agric..

[46]  Lars Grimstad,et al.  Development and field evaluation of a strawberry harvesting robot with a cable-driven gripper , 2019, Comput. Electron. Agric..

[47]  Won Suk Lee,et al.  A Machine Vision System for Quantification of Citrus Fruit Dropped on the Ground under the Canopy , 2015 .

[48]  Jørgen Kongsro,et al.  Estimation of pig weight using a Microsoft Kinect prototype imaging system , 2014 .

[49]  Markus Vincze,et al.  An Empirical Evaluation of Ten Depth Cameras: Bias, Precision, Lateral Noise, Different Lighting Conditions and Materials, and Multiple Sensor Setups in Indoor Environments , 2019, IEEE Robotics & Automation Magazine.

[50]  Matteo Munaro,et al.  OpenPTrack: Open source multi-camera calibration and people tracking for RGB-D camera networks , 2016, Robotics Auton. Syst..

[51]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[52]  Changying Li,et al.  Image processing algorithms for infield single cotton boll counting and yield prediction , 2019, Comput. Electron. Agric..

[53]  Jonathan Avendaño,et al.  Measurement of the ripening rate on coffee branches by using 3D images in outdoor environments , 2018, Comput. Ind..

[54]  Tristan Perez,et al.  Autonomous Sweet Pepper Harvesting for Protected Cropping Systems , 2017, IEEE Robotics and Automation Letters.

[55]  Ja-Ling Wu,et al.  Range data acquisition using color structured lighting and stereo vision , 1997, Image Vis. Comput..

[56]  Jordi Llorens,et al.  Kinect v2 Sensor-Based Mobile Terrestrial Laser Scanner for Agricultural Outdoor Applications , 2017, IEEE/ASME Transactions on Mechatronics.

[57]  Zhao Zhang,et al.  Wheat Lodging Detection from UAS Imagery Using Machine Learning Algorithms , 2020, Remote. Sens..

[58]  Elling Ruud Øye,et al.  GRIBBOT - Robotic 3D vision-guided harvesting of chicken fillets , 2016, Comput. Electron. Agric..

[59]  Volkan Isler,et al.  A comparative study of fruit detection and counting methods for yield mapping in apple orchards , 2018, J. Field Robotics.

[60]  Ke Wang,et al.  LSSA_CAU: An interactive 3d point clouds analysis software for body measurement of livestock with similar forms of cows or pigs , 2017, Comput. Electron. Agric..

[61]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[62]  Renfu Lu,et al.  A review of bin filling technologies for apple harvest and postharvest handling , 2018 .

[63]  F. García-Luna,et al.  Towards an artificial vision-robotic system for tomato identification , 2016 .

[64]  Tristan Perez,et al.  Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting—Combined Color and 3-D Information , 2017, IEEE Robotics and Automation Letters.

[65]  Xin Zhang,et al.  Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN , 2020, Comput. Electron. Agric..

[66]  Grzegorz Cielniak,et al.  3D-vision based detection, localization, and sizing of broccoli heads in the field , 2017, J. Field Robotics.

[67]  Jing Zhang,et al.  Multi-class object detection using faster R-CNN and estimation of shaking locations for automated shake-and-catch apple harvesting , 2020, Comput. Electron. Agric..

[68]  Bin Li,et al.  Using color and 3D geometry features to segment fruit point cloud and improve fruit recognition accuracy , 2020, Comput. Electron. Agric..

[69]  Wolfgang Junge,et al.  Automated calculation of udder depth and rear leg angle in Holstein-Friesian cows using a multi-Kinect cow scanning system , 2017 .

[70]  Chris McCarthy,et al.  Deep learning - Method overview and review of use for fruit detection and yield estimation , 2019, Comput. Electron. Agric..

[71]  Arno Ruckelshausen,et al.  An automatic and rapid system for grading palm bunch using a Kinect camera , 2017, Comput. Electron. Agric..

[72]  Dongjian He,et al.  Deep Learning Approach for Apple Edge Detection to Remotely Monitor Apple Growth in Orchards , 2020, IEEE Access.

[73]  Changying Li,et al.  Size estimation of sweet onions using consumer-grade RGB-depth sensor , 2014 .

[74]  R. C. Harrell,et al.  Robotic picking of citrus , 1990, Robotica.

[75]  Giulio Reina,et al.  In-field high throughput grapevine phenotyping with a consumer-grade depth camera , 2021, Comput. Electron. Agric..

[76]  V. Alchanatis,et al.  Low and high-level visual feature-based apple detection from multi-modal images , 2010, Precision Agriculture.

[77]  Xiangjun Zou,et al.  Color-, depth-, and shape-based 3D fruit detection , 2019, Precision Agriculture.

[78]  Lie Tang,et al.  Crop recognition under weedy conditions based on 3D imaging for robotic weed control , 2018, J. Field Robotics.

[79]  D. Stajnko,et al.  Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging , 2004 .

[80]  Ludovico Minto,et al.  Time-of-Flight and Structured Light Depth Cameras , 2016, Springer International Publishing.

[81]  Z. Shi,et al.  A robust method to estimate foliar phosphorus of rubber trees with hyperspectral reflectance , 2018, Industrial Crops and Products.

[82]  Roemi Fernández,et al.  Multisensory System for Fruit Harvesting Robots. Experimental Testing in Natural Scenarios and with Different Kinds of Crops , 2014, Sensors.

[83]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[84]  Jude Liu,et al.  The Development of Mechanical Apple Harvesting Technology: A Review , 2016 .

[85]  Chao Chen,et al.  Fast implementation of real-time fruit detection in apple orchards using deep learning , 2020, Comput. Electron. Agric..

[86]  Jun Zhou,et al.  Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking , 2017, Comput. Electron. Agric..

[87]  Rui Li,et al.  Kiwifruit detection in field images using Faster R-CNN with VGG16 , 2019, IFAC-PapersOnLine.

[88]  Verónica Vilaplana,et al.  Multi-modal deep learning for Fuji apple detection using RGB-D cameras and their radiometric capabilities , 2019, Comput. Electron. Agric..

[89]  Jing Pang,et al.  Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images , 2020, Precision Agriculture.