Faster R-CNN for multi-class fruit detection using a robotic vision system

Abstract An accurate and real-time image based multi-class fruit detection system is important for facilitating higher level smart farm tasks such as yield mapping and robotic harvesting. Robotic harvesting can reduce the costs of labour and increase fruit quality. This paper proposes a deep learning framework for multi-class fruits detection based on improved Faster R-CNN. The proposed framework includes fruits image library creation, data argumentation, improved Faster RCNN model generation, and performance evaluation. This work is a pioneer to create a multi-labeled and knowledge-based outdoor orchard image library using 4000 images in the real world. Also, improvement of the convolutional and pooling layers is achieved to have a more accurate and faster detection. The test results show the proposed algorithm has achieved higher detecting accuracy and lower processing time than the traditional detectors, which has excellent potential to build an autonomous and real-time harvesting or yield mapping/estimation system.

[1]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[2]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Ping Chen,et al.  On Improving the accuracy with Auto-Encoder on Conjunctivitis , 2019, Appl. Soft Comput..

[4]  James Patrick Underwood,et al.  Deep fruit detection in orchards , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Qiang Ni,et al.  Cognitive computing and wireless communications on the edge for healthcare service robots , 2020, Comput. Commun..

[6]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[7]  Sanjiv Singh,et al.  Automated Visual Yield Estimation in Vineyards , 2014, J. Field Robotics.

[8]  Arun Kumar Sangaiah,et al.  Image caption generation with high-level image features , 2019, Pattern Recognit. Lett..

[9]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[11]  Won Suk Lee,et al.  Immature green citrus detection based on colour feature and sum of absolute transformed difference (SATD) using colour images in the citrus grove , 2016, Comput. Electron. Agric..

[12]  Yuling Xi,et al.  Stimulus-driven and concept-driven analysis for image caption generation , 2020, Neurocomputing.

[13]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[16]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[17]  Mihai Oltean,et al.  Fruit recognition from images using deep learning , 2017, Acta Universitatis Sapientiae, Informatica.

[18]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[19]  Shaohua Wan,et al.  A unified two-parallel-branch deep neural network for joint gland contour and segmentation learning , 2019, Future Gener. Comput. Syst..

[20]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Deyu Wang,et al.  Cognitive-inspired class-statistic matching with triple-constrain for camera free 3D object retrieval , 2019, Future Gener. Comput. Syst..

[22]  Yu Zhao,et al.  Knowledge-Aided Convolutional Neural Network for Small Organ Segmentation , 2019, IEEE Journal of Biomedical and Health Informatics.

[23]  Chen Chen,et al.  Deep Learning and Superpixel Feature Extraction Based on Contractive Autoencoder for Change Detection in SAR Images , 2018, IEEE Transactions on Industrial Informatics.

[24]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[26]  Wolfram Burgard,et al.  Multimodal deep learning for robust RGB-D object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).