Instance Segmentation and Number Counting of Grape Berry Images Based on Deep Learning

In order to achieve accurate segmentation of each grape image per berry, we construct a dataset composed of red globe grape samples and use a two-stage “localization–segmentation” framework-based mask region convolutional neural network (Mask R-CNN) and one-stage “pixel classification without localization” framework-based You Only Look At CoefficienTs (YOLACT) and segmenting objects by locations (SOLO) models in the grape segmentation experiments. The optimal performance of the model Mask R-CNN was applied for further study. To address the problem of overlapping and occlusion causing inaccurate fruit detection in this model, the postprocessing algorithm of the Mask R-CNN model was improved by using the linear weighting method, and the experimental results were significantly improved. The model average precision (AP)0.50, AP0.75, the mean average precision (mAP), and the mean intersection of union (mIoU) improved by 1.98%, 2.72%, 4.30%, and 3.55%, respectively. The correlation coefficient was improved from 93.59% to 96.13% by using the improved Mask R-CNN to count the number of red globe grape berries, which also further illustrates that the fruit detection problem was well solved. Using the generalized method on untrained images of different grape varieties in different scenes also achieved good segmentation results. In this study, we provide a method for segmenting and counting grape berries that is useful for automating the grape industry.

[1]  Wang Yijie,et al.  A segmentation method for waxberry image under orchard environment , 2020 .

[2]  Yuyu Tian,et al.  Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot , 2020, Comput. Electron. Agric..

[3]  Ribana Roscher,et al.  Counting of Grapevine Berries in Images via Semantic Segmentation using Convolutional Neural Networks , 2020, ArXiv.

[4]  Yuning Jiang,et al.  SOLO: Segmenting Objects by Locations , 2019, ECCV.

[5]  Eleni Vrochidou,et al.  Grapes Visual Segmentation for Harvesting Robots Using Local Texture Descriptors , 2019, ICVS.

[6]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[7]  Thiago T. Santos,et al.  Grape detection, segmentation and tracking using deep neural networks and three-dimensional association , 2019, Comput. Electron. Agric..

[8]  Yong Jae Lee,et al.  YOLACT: Real-Time Instance Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[9]  Mohammad Reza Maleki,et al.  A robust algorithm based on color features for grape cluster segmentation , 2017, Comput. Electron. Agric..

[10]  Larry S. Davis,et al.  Soft-NMS — Improving Object Detection with One Line of Code , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[12]  Serge J. Belongie,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Stephen Nuske,et al.  Texture-based fruit detection via images using the smooth patterns on the fruit , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[18]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Dani Martínez,et al.  Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination , 2014 .

[20]  Sanjiv Singh,et al.  Automated Visual Yield Estimation in Vineyards , 2014, J. Field Robotics.

[21]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[22]  Salviano F. S. P. Soares,et al.  Automatic detection of bunches of grapes in natural environment from color images , 2012, J. Appl. Log..

[23]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[24]  Fumiomi Takeda,et al.  Three-dimensional photogrammetry with deep learning instance segmentation to extract berry fruit harvestability traits , 2021 .

[25]  MA Ben-xu Study on the Recognition Method of Grape in Different Natural Environment , 2015 .