Extracting apple tree crown information from remote imagery using deep learning

Abstract Manual measurement and visual inspection is a common practice for acquiring crop data in orchards and is a labor-intensive, time-consuming, and costly task. Accurate and rapid acquisition of crop data is vital for monitoring the dynamics of tree growth and optimizing farm management. In this work, we present a technique for orchard data acquisition and analysis that uses remote imagery acquired from unmanned aerial vehicles (UAVs) combined with deep learning convolutional neural networks to automatically detect and segment individual trees and measure the crown width, perimeter, and crown projection area of apple trees. By using an UAV platform, 50 high-resolution images of apple trees were collected from an orchard during dormancy (bare branches), and then each apple tree was detected by using a Faster R-CNN object detector. Based on these results, each tree was segmented by using a U-Net deep learning network. After convex tree boundaries were extracted from the semantic segmentation results by using an efficient pruning strategy, the crown parameters were automatically calculated, and the accuracy was compared with that obtained by manual delineation. The results show that the proposed remote sensing technique can be used to detect and count apple trees with precision and recall of 91.1% and 94.1%, respectively, segment their branches with an overall accuracy of 97.1%, and estimate crown parameter with an overall accuracy exceeding 92%. We conclude that this method not only saves labor by avoiding field measurements but also allows growers to dynamically monitor the growth of orchard trees.

[1]  Qingjie Liu,et al.  Road Extraction by Deep Residual U-Net , 2017, IEEE Geoscience and Remote Sensing Letters.

[2]  Xiaogang Wang,et al.  Cross-scene crowd counting via deep convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Stefan W. Maier,et al.  Efficiency of Individual Tree Detection Approaches Based on Light-Weight and Low-Cost UAS Imagery in Australian Savannas , 2018, Remote. Sens..

[4]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[5]  Changying Li,et al.  High Throughput Phenotyping of Blueberry Bush Morphological Traits Using Unmanned Aerial Systems , 2017, Remote. Sens..

[6]  Wang Hechun,et al.  Survey of Deep Learning Based Object Detection , 2019, Proceedings of the 2nd International Conference on Big Data Technologies - ICBDT2019.

[7]  Emmanuelle Gouillart,et al.  scikit-image: image processing in Python , 2014, PeerJ.

[8]  K. Walsh,et al.  Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’ , 2019, Precision Agriculture.

[9]  Thomas Brox,et al.  U-Net: deep learning for cell counting, detection, and morphometry , 2018, Nature Methods.

[10]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[11]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[12]  Weijia Li,et al.  Deep Learning Based Oil Palm Tree Detection and Counting for High-Resolution Remote Sensing Images , 2016, Remote. Sens..

[13]  Seishi Ninomiya,et al.  Characterization of peach tree crown by using high-resolution images from an unmanned aerial vehicle , 2018, Horticulture Research.

[14]  F. Frances Yao,et al.  Finding the Convex Hull of a Simple Polygon , 1983, J. Algorithms.

[15]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[16]  Maggi Kelly,et al.  Identification of Citrus Trees from Unmanned Aerial Vehicle Imagery Using Convolutional Neural Networks , 2018, Drones.

[17]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Yiannis Ampatzidis,et al.  Citrus rootstock evaluation utilizing UAV-based remote sensing and artificial intelligence , 2019, Comput. Electron. Agric..

[19]  Shang Gao,et al.  Deep Learning: Individual Maize Segmentation From Terrestrial Lidar Data Using Faster R-CNN and Regional Growth Algorithms , 2018, Front. Plant Sci..

[20]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[21]  Yiannis Ampatzidis,et al.  UAV-Based High Throughput Phenotyping in Citrus Utilizing Multispectral Imaging and Artificial Intelligence , 2019, Remote. Sens..

[22]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[23]  Saeid Minaei,et al.  Ultrasonic sensing of pistachio canopy for low-volume precision spraying , 2015, Comput. Electron. Agric..

[24]  M. Pascual,et al.  An Image-based Method to Study the Fruit Tree Canopy and the Pruning Biomass Production in a Peach Orchard , 2015 .

[25]  Margaret Lech,et al.  Evaluating deep learning architectures for Speech Emotion Recognition , 2017, Neural Networks.

[26]  Avadesh Meduri,et al.  MangoNet: A deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard , 2019, Eng. Appl. Artif. Intell..

[27]  Vijayan K. Asari,et al.  Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation , 2018, ArXiv.

[28]  Ronald L. Graham,et al.  An Efficient Algorithm for Determining the Convex Hull of a Finite Planar Set , 1972, Inf. Process. Lett..

[29]  Chao Wang,et al.  Multi-Temporal SAR Data Large-Scale Crop Mapping Based on U-Net Model , 2019, Remote. Sens..

[30]  James Patrick Underwood,et al.  Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry , 2016, Sensors.

[31]  Terry Griffin,et al.  Early-Season Stand Count Determination in Corn via Integration of Imagery from Unmanned Aerial Systems (UAS) and Supervised Learning Techniques , 2018, Remote. Sens..

[32]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[33]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[34]  Liang Han,et al.  Automatic Counting of in situ Rice Seedlings from UAV Images Based on a Deep Fully Convolutional Neural Network , 2019, Remote. Sens..

[35]  H. E. Ozkan,et al.  Development of a Variable-Rate Sprayer with Laser Scanning Sensor to Synchronize Spray Outputs to Tree Structures , 2012 .

[36]  Alexey Shvets,et al.  Fully Convolutional Network for Automatic Road Extraction from Satellite Imagery , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Josef Kittler,et al.  Minimum error thresholding , 1986, Pattern Recognit..

[38]  Giulio Reina,et al.  A Survey of Ranging and Imaging Techniques for Precision Agriculture Phenotyping , 2017, IEEE/ASME Transactions on Mechatronics.

[39]  P. Surový,et al.  Determining tree height and crown diameter from high-resolution UAV imagery , 2017 .

[40]  Joaquim Salvi,et al.  Multiple Sclerosis Lesion Synthesis in MRI Using an Encoder-Decoder U-NET , 2019, IEEE Access.

[41]  Terje Gobakken,et al.  Inventory of Small Forest Areas Using an Unmanned Aerial System , 2015, Remote. Sens..

[42]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Zhiguo Cao,et al.  TasselNet: counting maize tassels in the wild via local counts regression network , 2017, Plant Methods.

[44]  En Li,et al.  Apple detection during different growth stages in orchards using the improved YOLO-V3 model , 2019, Comput. Electron. Agric..