Detection and counting of flowers on apple trees for better chemical thinning decisions

Accurate chemical thinning of apple trees requires estimation of their blooming intensity, and determination of the blooming peak date. Performing this task, as of today, requires human experts to be present in the orchards for the entire blossom period or extrapolate using a single observation. Since experts are rare and in high demand, there is a need to automate this process. The system presented in this paper is able to estimate the blooming intensity and the blooming peak date from a sequence of tree images, with close-to-human accuracy. For this purpose, a two years dataset was collected in 2014–2015, partially tagged for the flowers location and completely annotated for blooming intensity. Using this dataset, an algorithm was developed and trained with three stages: a visual flower detector based on a deep convolutional neural network, followed by a blooming level estimator, and a peak blooming day finding algorithm. Despite the challenging conditions, the trained detector was able to detect flowers on trees with an Average Precision (AP) score of 0.68, which is on a par with contemporary results of other objects in detection benchmarks. The blooming estimator was based on a linear regression component, which used the number of flowers detected and related statistics to estimate the blooming intensity. The Pearson correlation between the algorithm blooming estimation and human judgments of several experts indicated high agreement levels (0.78–0.93) which were similar to the correlations measured among the human experts. Moreover, the developed estimator was relatively stable across multiple years. The developed peak date finding algorithm identified correctly the orchard’s blooming peak date, which was used to determine the thinning date in the current practice (the entire orchard is thinned in the same day). Experiments testing the algorithm’s ability to find a blooming peak date for each tree independently showed encouraging results, which may lead upon refinement to a more precise practice for tree-specific thinning.

[1]  Qi Wang,et al.  Automated Crop Yield Estimation for Apple Orchards , 2012, ISER.

[2]  Hanno Scharr,et al.  Finely-grained annotated datasets for image-based plant phenotyping , 2016, Pattern Recognit. Lett..

[3]  Sotirios A. Tsaftaris,et al.  Leveraging multiple datasets for deep leaf counting , 2017, bioRxiv.

[4]  Daniele Nardi,et al.  Fast and Accurate Crop and Weed Identification with Summarized Train Sets for Precision Agriculture , 2016, IAS.

[5]  A. Lakso,et al.  Advances in Predicting Chemical Thinner Response of Apple Using a Carbon Balance Model , 2012 .

[6]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[7]  R. Byers,et al.  Effects of Temperature and the Combination of Liquid Lime Sulfur and Fish Oil on Pollen Germination, Pollen Tube Growth, and Fruit Set in Apples , 2009 .

[8]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[9]  Raphael Linker,et al.  Determination of the number of green apples in RGB images recorded in orchards , 2012 .

[10]  James Patrick Underwood,et al.  Deep fruit detection in orchards , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Ashok Samal,et al.  Image-Based Plant Phenotyping : Opportunities and Challenges , 2020 .

[12]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Brane Širok,et al.  Flowering estimation in apple orchards by image analysis , 2013, Precision Agriculture.

[14]  Bernt Schiele,et al.  Learning Non-maximum Suppression , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Billy V. Koen,et al.  Definition of the engineering method , 1985 .

[16]  F. G. Jr. Dennis,et al.  The history of fruit thinning , 2000, Plant Growth Regulation.

[17]  Yael Edan,et al.  Computer vision for fruit harvesting robots - state of the art and challenges ahead , 2012, Int. J. Comput. Vis. Robotics.

[18]  Chester Forshey,et al.  CHEMICAL FRUIT THINNING OF APPLES , 1986 .

[19]  H. Link,et al.  Significance of flower and fruit thinning on fruit quality , 2000, Plant Growth Regulation.

[20]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[21]  Edward M. Barnes,et al.  Method for Using Images from a Color Digital Camera to Estimate Flower Number , 2000 .

[22]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[23]  Marcel Salathé,et al.  Using Deep Learning for Image-Based Plant Disease Detection , 2016, Front. Plant Sci..

[24]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[25]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[26]  D. Bochtis,et al.  Yield prediction in apple orchards based on image processing , 2011, Precision Agriculture.

[27]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.