Colour-agnostic shape-based 3D fruit detection for crop harvesting robots

Most agricultural robots, fruit harvesting systems in particular, use computer vision to detect their fruit targets. Exploiting the uniqueness of fruit colour amidst the foliage, almost all of these computer vision systems rely on colour features to identify the fruit in the image. However, often the colour of fruit cannot be discriminated from its background, especially under unstable illumination conditions, thus rendering the detection and segmentation of the target highly sensitive or unfeasible in colour space. While multispectral signals, especially those outside the visible spectrum, may alleviate this difficulty, simpler, cheaper, and more accessible solutions are desired. Here exploiting both RGB and range data to analyse shape-related features of objects both in the image plane and 3D space is proposed. In particular, 3D surface normal features, 3D plane-reflective symmetry, and image plane highlights from elliptic surface points are combined to provide shape-based detection of fruits in 3D space regardless of their colour. Results are shown using a particularly challenging sweet pepper dataset with a significant degree of occlusions.

[1]  Shree K. Nayar,et al.  Separation of Reflection Components Using Color and Polarization , 1997, International Journal of Computer Vision.

[2]  Sebastian Thrun,et al.  Shape from symmetry , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[3]  Joe Rosen.  Symmetry Discovered: Concepts and Applications in Nature and Science , 1975 .

[4]  Vincent Lepetit,et al.  Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes , 2011, 2011 International Conference on Computer Vision.

[5]  James M. Keller,et al.  Histogram of Oriented Normal Vectors for Object Recognition with a Depth Sensor , 2012, ACCV.

[6]  Kai Oliver Arras,et al.  People detection in RGB-D data , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[7]  Ohad Ben-Shahar,et al.  Depth Based Object Detection from Partial Pose Estimation of Symmetric Objects , 2014, ECCV.

[8]  Yael Edan,et al.  Robotic melon harvesting , 2000, IEEE Trans. Robotics Autom..

[9]  Katsushi Ikeuchi,et al.  Reflection components decomposition of textured surfaces using linear basis functions , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[10]  Jan-Olof Eklundh,et al.  Detecting Symmetry and Symmetric Constellations of Features , 2006, ECCV.

[11]  Margarita Osadchy,et al.  Using specular highlights as pose invariant features for 2D-3D pose estimation , 2011, CVPR 2011.

[12]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[13]  Nico Blodow,et al.  CAD-model recognition and 6DOF pose estimation using 3D cues , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[14]  P. Beckmann,et al.  The scattering of electromagnetic waves from rough surfaces , 1963 .

[15]  David J. Kriegman,et al.  Specularity Removal in Images and Videos: A PDE Approach , 2006, ECCV.

[16]  Stephen Lin,et al.  Separation of diffuse and specular reflection in color images , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[17]  R. C. Harrell,et al.  A fruit-tracking system for robotic harvesting , 2005, Machine Vision and Applications.

[18]  Saturnino Maldonado-Bascón,et al.  SURFing the point clouds: Selective 3D spatial pyramids for category-level object recognition , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Sang Wook Lee,et al.  Detection of Specularity Using Color and Multiple Views , 1992, ECCV.

[20]  M. Monta,et al.  Three-dimensional sensing system for agricultural robots , 2003, Proceedings 2003 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM 2003).

[21]  Hui-Liang Shen,et al.  Simple and efficient method for specularity removal in an image. , 2009, Applied optics.

[22]  Long Quan,et al.  Translation symmetry detection in a fronto-parallel view , 2011, CVPR 2011.

[23]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[24]  David J. Kriegman,et al.  Beyond Lambert: reconstructing specular surfaces using color , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[25]  Dieter Fox,et al.  Object recognition with hierarchical kernel descriptors , 2011, CVPR 2011.

[26]  Dieter Fox,et al.  Detection-based object labeling in 3D scenes , 2012, 2012 IEEE International Conference on Robotics and Automation.

[27]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Sang Wook Lee,et al.  Detection of specularity using colour and multiple views , 1992, Image Vis. Comput..

[29]  David A. McAllester,et al.  A discriminatively trained, multiscale, deformable part model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Ohad Ben-Shahar,et al.  Shape from specular flow: Is one flow enough? , 2011, CVPR 2011.

[31]  Niloy J. Mitra,et al.  Symmetry in 3D Geometry: Extraction and Applications , 2013, Comput. Graph. Forum.

[32]  Dieter Fox,et al.  A large-scale hierarchical multi-view RGB-D object dataset , 2011, 2011 IEEE International Conference on Robotics and Automation.

[33]  José Luis Pons Rovira,et al.  Machine Vision and Applications Manuscript-nr. a Vision System Based on a Laser Range--nder Applied to Robotic Fruit Harvesting , 2022 .

[34]  Peter P. Ling,et al.  Fast Fruit identification for Robotic Tomato Picker , 2004 .

[35]  David W. Jacobs,et al.  Using specularities for recognition , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[36]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[37]  Yanxi Liu,et al.  Performance evaluation of state-of-the-art discrete symmetry detection algorithms , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Jose L Pons,et al.  A SURVEY OF COMPUTER VISION METHODS FOR LOCATING FRUIT ON TREES , 2000 .

[39]  Trevor Darrell,et al.  Practical 3-D Object detection using category and instance-level appearance models , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[40]  Ohad Ben-Shahar,et al.  Shape from Specular Flow , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[42]  Stephen Lin,et al.  Diffuse-Specular Separation and Depth Recovery from Image Sequences , 2002, ECCV.

[43]  M. Wertheimer.  Untersuchungen zur Lehre von der Gestalt. II , 1923 .

[44]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[45]  Katsushi Ikeuchi,et al.  Temporal-color space analysis of reflection , 1994 .

[46]  Silvio Savarese,et al.  Accurate Localization of 3D Objects from RGB-D Data Using Segmentation Hypotheses , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Stephen Lin,et al.  Separation of Highlight Reflections on Textured Surfaces , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[48]  Niloy J. Mitra,et al.  Symmetry in 3D Geometry: Extraction and Applications , 2013, Comput. Graph. Forum.

[49]  Max Wertheimer,et al.  Untersuchungen zur Lehre von der Gestalt , .

[50]  Larry S. Davis,et al.  A Pose-Invariant Descriptor for Human Detection and Segmentation , 2008, ECCV.

[51]  Haim Levkowitz,et al.  Color Theory and Modeling for Computer Graphics, Visualization, and Multimedia Applications , 1997 .

[52]  Nathan Silberman,et al.  Indoor scene segmentation using a structured light sensor , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[53]  Jitendra Malik,et al.  Blobworld: A System for Region-Based Image Indexing and Retrieval , 1999, VISUAL.

[54]  Jonathan T. Barron,et al.  A category-level 3-D object dataset: Putting the Kinect to work , 2011, ICCV Workshops.

[55]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[56]  Katsushi Ikeuchi,et al.  Separating Reflection Components of Textured Surfaces Using a Single Image , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[57]  Ohad Ben-Shahar,et al.  Specular Flow and Shape in One Shot: , 2011, BMVC.

[58]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[59]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[60]  Jong-Seok Park,et al.  Highlight separation and surface orientations for 3-D specular objects , 1990, [1990] Proceedings. 10th International Conference on Pattern Recognition.

[61]  T. F. Burks,et al.  Current Developments in Automated Citrus Harvesting , 2004 .

[62]  Yanxi Liu,et al.  Curved glide-reflection symmetry detection , 2009, CVPR.