A deep learning system for single and overall weight estimation of melons using unmanned aerial vehicle images

Abstract Generation of yield maps enables making agronomic decisions related to resource management and marketing, leading to improved production and breeding processes. Estimating melon yield production before harvest at single-melon resolution is a labor-intensive task, requiring a detailed account of accumulated yield and general yield distribution, as well as detailed measurements of melon size and location. This study presents an algorithmic pipeline for detection and yield estimation of melons from top-view color images acquired by a digital camera mounted on an unmanned aerial vehicle. The yield estimation provides both the number of melons and the weight of each melon. The system includes three main stages: melon detection, geometric feature extraction, and individual melon yield estimation. The melon-detection process was based on the RetinaNet deep convolutional neural network. Transfer learning was used for the training to detect small objects in high-resolution images successfully. The detection process achieved an average precision score of 0.92 with a F1 score of more than 0.9 in a variety of agricultural environments. For each detected melon, feature extraction was applied using the Chan–Vese active contour algorithm and principal component analysis ellipse-fitting method. A regression model that ties the ellipse features to the melon’s weight is presented. The modified (adjusted) R Adj 2 value of the regression model was 0.94. The system results for estimating the weight of a single melon measured by the mean absolute percentage error index achieved 16%. The analysis revealed that this could be decreased to 12% error with more accurate geometrical feature extraction. Overall yield estimation derived by summing the weights of all melons in the field resulted in only a 3% underestimation of the actual total yield.

[1]  Vijay Kumar,et al.  Counting Apples and Oranges With Deep Learning: A Data-Driven Approach , 2017, IEEE Robotics and Automation Letters.

[2]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[3]  Mark R. Pickering,et al.  Food Volume Estimation in a Mobile Phone Based Dietary Assessment System , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[4]  Rafael Rieder,et al.  Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review , 2018, Comput. Electron. Agric..

[5]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Hong Cheng,et al.  Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks , 2017, J. Imaging.

[7]  Marios Anthimopoulos,et al.  Two-View 3D Reconstruction for Food Volume Estimation , 2017, IEEE Transactions on Multimedia.

[8]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[9]  Chris McCarthy,et al.  Deep learning - Method overview and review of use for fruit detection and yield estimation , 2019, Comput. Electron. Agric..

[10]  James Patrick Underwood,et al.  Deep fruit detection in orchards , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[12]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[13]  Tony F. Chan,et al.  An Active Contour Model without Edges , 1999, Scale-Space.

[14]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[15]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[16]  Pascual Campoy Cervera,et al.  A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles , 2017, J. Sensors.

[17]  Edward J. Delp,et al.  Food image analysis: Segmentation, identification and weight estimation , 2013, 2013 IEEE International Conference on Multimedia and Expo (ICME).

[18]  Tarique da Silveira Cavalcante,et al.  A computer vision model development for size and weight estimation of yellow melon in the Brazilian northeast , 2019, Scientia Horticulturae.

[19]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[20]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[21]  Avadesh Meduri,et al.  MangoNet: A deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard , 2019, Eng. Appl. Artif. Intell..

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[25]  Pablo M. Granitto,et al.  Deep learning for plant identification using vein morphological patterns , 2016, Comput. Electron. Agric..

[26]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[28]  Raul Morais,et al.  Recent advances in image processing techniques for automated harvesting purposes: A review , 2017, 2017 Intelligent Systems Conference (IntelliSys).

[29]  Mooi Choo Chuah,et al.  A Strawberry Detection System Using Convolutional Neural Networks , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[30]  Shervin Shirmohammadi,et al.  Measuring Calorie and Nutrition From Food Image , 2014, IEEE Transactions on Instrumentation and Measurement.

[31]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[32]  Jeff G. Schneider,et al.  Flexible Transfer Learning under Support and Model Shift , 2014, NIPS.

[33]  Victor Alchanatis,et al.  Detection and counting of flowers on apple trees for better chemical thinning decisions , 2019, Precision Agriculture.

[34]  Eldert J. van Henten,et al.  Simulation of harvest operations in a static rose cultivation system , 2014 .

[35]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[36]  Won Suk Lee,et al.  Strawberry Yield Prediction Based on a Deep Neural Network Using High-Resolution Aerial Orthoimages , 2019, Remote. Sens..

[37]  Cyrill Stachniss,et al.  REAL-TIME BLOB-WISE SUGAR BEETS VS WEEDS CLASSIFICATION FOR MONITORING FIELDS USING CONVOLUTIONAL NEURAL NETWORKS , 2017 .

[38]  Stefano Cagnoni,et al.  A New Approach to Image-Based Estimation of Food Volume , 2017, Algorithms.

[39]  D. Stajnko,et al.  Modelling apple fruit yield using image analysis for fruit colour, shape and texture. , 2009 .

[40]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[41]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[42]  Kushtrim Bresilla,et al.  Single-Shot Convolution Neural Networks for Real-Time Fruit Detection Within the Tree , 2019, Front. Plant Sci..

[43]  Wei Sun,et al.  A Real-Time Detection Framework for On-Tree Mango Based on SSD Network , 2018, ICIRA.

[44]  K. Walsh,et al.  Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’ , 2019, Precision Agriculture.

[45]  Claire Anderson,et al.  Tomato Fruit Shape Analysis Using Morphometric and Morphology Attributes Implemented in Tomato Analyzer Software Program , 2009 .

[46]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[47]  Patrizia Busato,et al.  Machine Learning in Agriculture: A Review , 2018, Sensors.

[48]  Yael Edan,et al.  Computer vision system for measuring individual cow feed intake using RGB-D camera and deep learning algorithms , 2020, Comput. Electron. Agric..

[49]  Andrew Robson,et al.  Exploring the Potential of High Resolution WorldView-3 Imagery for Estimating Yield of Mango , 2018, Remote. Sens..