An automated fruit harvesting robot by using deep learning

Automation and labor saving in agriculture have been required recently. However, mechanization and robots for growing fruits have not been advanced. This study proposes a method of detecting fruits and automated harvesting using a robot arm. A highly fast and accurate method with a Single Shot MultiBox Detector is used herein to detect the position of fruit, and a stereo camera is used to detect the three-dimensional position. After calculating the angles of the joints at the detected position by inverse kinematics, the robot arm is moved to the target fruit’s position. The robot then harvests the fruit by twisting the hand axis. The experimental results showed that more than 90% of the fruits were detected. Moreover, the robot could harvest a fruit in 16 s.

[1]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[2]  Liu Bin,et al.  Identification of fruit and branch in natural scenes for citrus harvesting robot using machine vision and support vector machine , 2014 .

[3]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[5]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[6]  Jean-Jacques E. Slotine,et al.  Robot analysis and control , 1988, Autom..

[7]  W. S. Lee,et al.  Green citrus detection using hyperspectral imaging , 2009 .

[8]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  D. Bulanon,et al.  A Segmentation Algorithm for the Automatic Recognition of Fuji Apples at Harvest , 2002 .

[10]  Raphael Linker,et al.  Determination of the number of green apples in RGB images recorded in orchards , 2012 .

[11]  D. Stajnko,et al.  Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging , 2004 .

[12]  S. Hata,et al.  Development of a real-time machine vision system for the apple harvesting robot , 2004, SICE 2004 Annual Conference.

[13]  Ferhat KurtulmusWon.  Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network , 2014 .

[14]  Takashi Kataoka,et al.  Fruit detection system and an end effector for robotic harvesting of Fuji apples , 2010 .

[15]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[16]  Raphael Linker,et al.  Estimation of the Number of Apples in Color Images Recorded in Orchards , 2010, CCTA.