A nearest neighbor approach for fruit recognition in RGB-D images based on detection of convex surfaces

Abstract Automatic fruit picking is a challenging problem in robotics with a wide application field. A prerequisite for realization of a robotic fruit picker is its ability to detect fruits in tree tops. An expert system, which would be able to compete with human perception, must be capable of recognizing fruits among leaves and branches under uncontrolled conditions, where fruits are occluded and shaded. In this paper, a novel approach for fruit recognition in RGB-D images based on detection and classification of convex surfaces is proposed. The input RGB-D image is first segmented into convex surfaces by a region growing procedure. Each convex surface is then described by an appropriate descriptor and classified with the aid of the associated descriptor. A novel descriptor of approximately convex surfaces is proposed, which we named Convex Template Instance (CTI) descriptor. It is based on approximating surfaces by convex polyhedrons with quantized face orientations, where every polyhedron face corresponds to one descriptor component. Computation of the proposed descriptor is simple and can be performed very efficiently. The proposed CTI descriptor is compared to the SHOT descriptor, a standard descriptor for 3D point clouds. Two variants of the both CTI and SHOT descriptor are evaluated, a variant which uses color and a variant which does not. A k-nearest neighbor classifier is used to classify detected surfaces into two classes: fruit and other. The main advantage of the proposed expert system in comparison to other fruit recognition solutions is its computational efficiency, which is of great importance for its target application – an automatic fruit picker. The proposed approach is evaluated using a challenging dataset containing RGB-D images of four fruit sorts acquired under uncontrolled conditions, which has been made publicly available to the scientific community, allowing benchmarking of novel fruit recognition methods.

[1]  Jesús Raúl Martínez-Sandoval,et al.  Implementation of Hough transform for fruit image segmentation , 2012 .

[2]  R. C. Harrell,et al.  A fruit-tracking system for robotic harvesting , 2005, Machine Vision and Applications.

[3]  Juan Ramón Rico-Juan,et al.  Clustering-based k-nearest neighbor classification for large-scale data with neural codes representation , 2018, Pattern Recognit..

[4]  José Luis Pons Rovira,et al.  Automatic fruit recognition: a survey and new results using Range/Attenuation images , 1999, Pattern Recognit..

[5]  Ohad Ben-Shahar,et al.  Colour-agnostic shape-based 3D fruit detection for crop harvesting robots , 2016 .

[6]  W. S. Lee,et al.  Identification and determination of the number of immature green citrus fruit in a canopy under different ambient light conditions , 2014 .

[7]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[8]  Federico Tombari,et al.  A combined texture-shape descriptor for enhanced 3D feature matching , 2011, 2011 18th IEEE International Conference on Image Processing.

[9]  Yan Chen,et al.  Recognition of Mature Citrus in Natural Scene under the Occlusion Condition , 2014 .

[10]  Nong Sang,et al.  Detecting citrus fruits with highlight on tree based on fusion of multi-map , 2014 .

[11]  C. Glasbey,et al.  Automatic fruit recognition and counting from multiple images , 2014 .

[12]  Damir Filko,et al.  Point Cloud Segmentation to Approximately Convex Surfaces for Fruit Recognition , 2014 .

[13]  D. Stajnko,et al.  Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging , 2004 .

[14]  Victor Alchanatis,et al.  Image fusion of visible and thermal images for fruit detection. , 2009 .

[15]  Peteris Eizentals,et al.  3D pose estimation of green pepper fruit for automated harvesting , 2016, Comput. Electron. Agric..

[16]  Ying Zhang,et al.  Automatic recognition vision system guided for apple harvesting robot , 2012, Comput. Electr. Eng..

[17]  Zhang Yajing,et al.  Identification of fruit and branch in natural scenes for citrus harvesting robot using machine vision and support vector machine , 2014 .

[18]  Ribana Roscher,et al.  Automated Image Analysis Framework for the High-Throughput Determination of Grapevine Berry Sizes Using Conditional Random Fields , 2017, ArXiv.

[19]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[20]  Volker Steinhage,et al.  High-precision 3D detection and reconstruction of grapes from laser range data for efficient phenotyping based on supervised learning , 2017, Comput. Electron. Agric..

[21]  Raphael Linker,et al.  Vision-based localisation of mature apples in tree images using convexity , 2014 .

[22]  Dani Martínez,et al.  A Proposal for Automatic Fruit Harvesting by Combining a Low Cost Stereovision Camera and a Robotic Arm , 2014, Sensors.

[23]  José Blasco,et al.  A new method for pedicel/peduncle detection and size assessment of grapevine berries and other fruits by image analysis , 2014 .

[24]  Xiaofeng Zhu,et al.  Efficient kNN Classification With Different Numbers of Nearest Neighbors , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Tristan Perez,et al.  Peduncle Detection of Sweet Pepper for Autonomous Crop Harvesting—Combined Color and 3-D Information , 2017, IEEE Robotics and Automation Letters.

[26]  Won Suk Lee,et al.  Identifying blueberry fruit of different growth stages using natural outdoor color images , 2014 .

[27]  David G. Lowe,et al.  Scalable Nearest Neighbor Algorithms for High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Jose L Pons,et al.  A SURVEY OF COMPUTER VISION METHODS FOR LOCATING FRUIT ON TREES , 2000 .

[29]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[30]  Andrew W. Moore,et al.  New Algorithms for Efficient High-Dimensional Nonparametric Classification , 2006, J. Mach. Learn. Res..

[31]  Heiner Kuhlmann,et al.  Surface feature based classification of plant organs from 3D laserscanned point clouds for plant phenotyping , 2013, BMC Bioinformatics.

[32]  Yibin Ying,et al.  Recognition of clustered tomatoes based on binocular stereo vision , 2014 .

[33]  Xin Chen,et al.  Fast segmentation of range images into planar regions , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Damir Filko,et al.  Fast 2.5D Mesh Segmentation to Approximately Convex Surfaces , 2011, ECMR.

[35]  Francisco Herrera,et al.  kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big data , 2017, Knowl. Based Syst..

[36]  Kristian Kersting,et al.  Automated interpretation of 3D laserscanned point clouds for plant organ segmentation , 2015, BMC Bioinformatics.

[37]  D. Bulanon,et al.  A Segmentation Algorithm for the Automatic Recognition of Fuji Apples at Harvest , 2002 .

[38]  Zhen Liu,et al.  The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment , 2018 .

[39]  Markus Vincze,et al.  Ensemble of shape functions for 3D object classification , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[40]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[41]  Jun Zhou,et al.  Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking , 2017, Comput. Electron. Agric..

[42]  Federico Tombari,et al.  Unique Signatures of Histograms for Local Surface Description , 2010, ECCV.

[43]  Fernando Alfredo Auat Cheeín,et al.  Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry , 2017, Comput. Electron. Agric..