Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards

Robotic harvesting of fruits in orchards is a challenging task, since high density and overlapping of fruits and branches can heavily impact the success rate of robotic harvesting. Therefore, the vision system is demanded to provide comprehensive information of the working environment to guide the manipulator and gripping system to successful detach the target fruits. In this study, a deep learning based one-stage detector DaSNet-V2 is developed to perform the multi-task vision sensing in the working environment of apple orchards. DaSNet-V2 combines the detection and instance segmentation of fruits and semantic segmentation of branch into a single network architecture. Meanwhile, a light-weight backbone network LW-net is utilised in the DaSNet-V2 model to improve the computational efficiency of the model. In the experiment, DaSNet-V2 is tested and evaluated on the RGB-D images of the orchard. From the experiment results, DaSNet-V2 with lightweight backbone achieves 0.844, 0.858, and 0.795 on the F 1 score of the detection, and mean intersection of union on the instance segmentation of fruits and semantic segmentation of branches, respectively. To provide a direct-viewing of the working environment in orchards, the obtained sensing results are illustrated by 3D visualisation . The robustness and efficiency of the DaSNet-V2 in detection and segmentation are validated by the experiments in the real-environment of apple orchard.

[1]  José García Rodríguez,et al.  A Review on Deep Learning Techniques Applied to Semantic Segmentation , 2017, ArXiv.

[2]  James Patrick Underwood,et al.  Deep fruit detection in orchards , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Anup Vibhute,et al.  Applications of Image Processing in Agriculture: A Survey , 2012 .

[4]  H. Valle,et al.  Australian vegetable growing farms: an economic survey, 2011-12 and 2012-13. , 2014 .

[5]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[6]  Tristan Perez,et al.  Visual detection of occluded crop: For automated harvesting , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Katsuhiko Sakaue,et al.  Utilization of stereo disparity and optical flow information for the computer analysis of human interactions , 2003, Machine Vision and Applications.

[8]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[9]  Yuanshen Zhao,et al.  A review of key techniques of vision-based control for harvesting robot , 2016, Comput. Electron. Agric..

[10]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  En Li,et al.  Apple detection during different growth stages in orchards using the improved YOLO-V3 model , 2019, Comput. Electron. Agric..

[12]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Lorenzo Comba,et al.  Unsupervised detection of vineyards by 3D point-cloud UAV photogrammetry for precision agriculture , 2018, Comput. Electron. Agric..

[14]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[15]  Josse De Baerdemaeker,et al.  Detection of red and bicoloured apples on tree with an RGB-D camera , 2016 .

[16]  Dong Xu,et al.  Advanced Deep-Learning Techniques for Salient and Category-Specific Object Detection: A Survey , 2018, IEEE Signal Processing Magazine.

[17]  K. Walsh,et al.  Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’ , 2019, Precision Agriculture.

[18]  Dietrich Paulus,et al.  Semantic 3D Octree Maps based on Conditional Random Fields , 2013, MVA.

[19]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Zhou Yu,et al.  SPRNet: Single-Pixel Reconstruction for One-Stage Instance Segmentation , 2019, IEEE Transactions on Cybernetics.

[21]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[22]  Xiangjun Zou,et al.  Color-, depth-, and shape-based 3D fruit detection , 2019, Precision Agriculture.

[23]  Lining Sun,et al.  RGB-D-Based Pose Estimation of Workpieces with Semantic Segmentation and Point Cloud Registration , 2019, Sensors.

[24]  Chao Chen,et al.  Fruit Detection and Segmentation for Apple Harvesting Using Visual Sensor in Orchards , 2019, Sensors.

[25]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[26]  R. Zhou,et al.  Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield , 2012, Precision Agriculture.

[27]  Fernando Auat Cheein,et al.  Human–robot interaction in agriculture: A survey and current challenges , 2019, Biosystems Engineering.

[28]  Xiangjun Zou,et al.  Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field , 2019, Sensors.

[29]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[30]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[31]  Yael Edan,et al.  Harvesting Robots for High‐value Crops: State‐of‐the‐art Review and Challenges Ahead , 2014, J. Field Robotics.

[32]  Rajib Bandyopadhyay,et al.  Rapid Evaluation of Integral Quality and Safety of Surface and Waste Waters by a Multisensor System (Electronic Tongue) , 2019, Sensors.

[33]  Lihong Xu,et al.  Unsupervised segmentation of greenhouse plant images based on modified Latent Dirichlet Allocation , 2018, PeerJ.

[34]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[35]  Bolei Zhou,et al.  SegICP: Integrated deep semantic segmentation and pose estimation , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Xiaoyang Liu,et al.  The recognition of apple fruits in plastic bags based on block classification , 2017, Precision Agriculture.