Using YOLOv3 Algorithm with Pre- and Post-Processing for Apple Detection in Fruit-Harvesting Robot

A machine vision system for detecting apples in orchards was developed. The system was designed to be used in harvesting robots and is based on a YOLOv3 algorithm with special pre- and post-processing. The proposed pre- and post-processing techniques made it possible to adapt the YOLOv3 algorithm to be used in an apple-harvesting robot machine vision system, providing an average apple detection time of 19 ms with a share of objects being mistaken for apples at 7.8% and a share of unrecognized apples at 9.2%. Both the average detection time and error rates are less than in all known similar systems. The system can operate not only in apple-harvesting robots but also in orange-harvesting robots.

[1]  Michael J. Delwiche,et al.  Computer Vision to Locate Fruit on a Tree , 1985 .

[2]  Chao Chen,et al.  Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards , 2019, Comput. Electron. Agric..

[3]  Yuanshen Zhao,et al.  A review of key techniques of vision-based control for harvesting robot , 2016, Comput. Electron. Agric..

[4]  Lvwen Huang,et al.  Ripe Fuji Apple Detection Model Analysis in Natural Tree Canopy , 2012 .

[5]  F. E. Sistler,et al.  Robotics and intelligent machines in agriculture , 1987, IEEE J. Robotics Autom..

[6]  G. E. Miles,et al.  Fruit location in a partially occluded image , 1984 .

[7]  Yiliang Zeng,et al.  Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot , 2014 .

[8]  Raphael Linker,et al.  Vision-based localisation of mature apples in tree images using convexity , 2014 .

[9]  Shaohua Wan,et al.  Faster R-CNN for multi-class fruit detection using a robotic vision system , 2020, Comput. Networks.

[10]  Jose L Pons,et al.  Design and implementation of an aided fruit‐harvesting robot (Agribot) , 1998 .

[11]  Yuanshen Zhao,et al.  Robust Tomato Recognition for Robotic Harvesting Using Feature Images Fusion , 2016, Sensors.

[12]  Jing Zhang,et al.  Branch detection for apple trees trained in fruiting wall architecture using depth features and Regions-Convolutional Neural Network (R-CNN) , 2018, Comput. Electron. Agric..

[13]  Mike Duke,et al.  Investigation of Optimal Network Architecture for Asparagus Spear Detection in Robotic Harvesting , 2019, IFAC-PapersOnLine.

[14]  Rui Li,et al.  Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion , 2020, IEEE Access.

[15]  Richard O. Duda,et al.  Use of the Hough transformation to detect lines and curves in pictures , 1972, CACM.

[16]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.

[17]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[18]  Jordi Llorens,et al.  Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow , 2020, Comput. Electron. Agric..

[19]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Ying Zhang,et al.  Automatic recognition vision system guided for apple harvesting robot , 2012, Comput. Electr. Eng..

[21]  Avital Bechar,et al.  Agricultural robots for field operations: Concepts and components , 2016 .

[22]  Raphael Linker,et al.  Determination of the number of green apples in RGB images recorded in orchards , 2012 .

[23]  Yuyu Tian,et al.  Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot , 2020, Comput. Electron. Agric..

[24]  Manjunath V. Joshi,et al.  Fruit Detection using Improved Multiple Features based Algorithm , 2011 .

[25]  Victor Alchanatis,et al.  Image fusion of visible and thermal images for fruit detection. , 2009 .

[26]  Chao Chen,et al.  Fast implementation of real-time fruit detection in apple orchards using deep learning , 2020, Comput. Electron. Agric..

[27]  Jun Zhou,et al.  Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking , 2017, Comput. Electron. Agric..

[28]  Josef Kittler,et al.  A survey of the hough transform , 1988, Comput. Vis. Graph. Image Process..

[29]  En Li,et al.  Apple detection during different growth stages in orchards using the improved YOLO-V3 model , 2019, Comput. Electron. Agric..

[30]  V. Alchanatis,et al.  Low and high-level visual feature-based apple detection from multi-modal images , 2010, Precision Agriculture.

[31]  E. A. Parrish,et al.  Pictorial Pattern Recognition Applied to Fruit Harvesting , 1977 .

[32]  Won Suk Lee,et al.  Green citrus detection using 'eigenfruit', color and circular Gabor texture features under natural outdoor conditions , 2011 .

[33]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[34]  En Li,et al.  Detection of Apple Lesions in Orchards Based on Deep Learning Methods of CycleGAN and YOLOV3-Dense , 2019, J. Sensors.

[35]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Nong Sang,et al.  Detecting citrus fruits with highlight on tree based on fusion of multi-map , 2014 .

[37]  Liang Gong,et al.  Detecting tomatoes in greenhouse scenes by combining AdaBoost classifier and colour analysis , 2016 .

[38]  Mihai Oltean,et al.  Fruit recognition from images using deep learning , 2017, Acta Universitatis Sapientiae, Informatica.

[39]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[40]  Won Suk Lee,et al.  Immature green citrus fruit detection using color and thermal images , 2018, Comput. Electron. Agric..