Automated grapevine flower detection and quantification method based on computer vision and deep learning from on-the-go imaging using a mobile sensing platform under field conditions

Abstract Grape yield forecasting is a valuable economic and quality issue for the grape and wine industry. The number of flowers at bloom could be used as an early indicator towards crop forecast in viticulture. The purpose of this work was to develop a non-invasive method for grapevine flower counting by on-the-go image acquisition, using a combination of deep learning and computer vision technology. A mobile sensing platform was used at 5 km/h to automatically capture Red Green Blue (RGB) images of vineyard canopy at night using artificial illumination under field conditions. For the image data set, 96 vines from six grapevine varieties were selected. For ground-truthing, the number of flowers per inflorescence was counted on a set of clusters before flowering, while yield per vine was weighted at harvest. The developed algorithm comprised two general steps: inflorescences’ segmentation, and individual flower detection. In both steps, the best results were obtained using the deep fully convolutional neural network SegNet architecture with a VGG19 network as the encoder, with F1 score values of 0.93 and 0.73 in the inflorescences segmentation and the individual flower detection steps, respectively. These values showed the high accuracy of the network. A determination coefficient (R2) of 0.91 between the detected number of flowers and the actual number of flowers per vine was obtained. In addition, a linear regression model was trained to estimate the actual number of flowers from the number of detected flowers. A root mean squared error (RMSE) of 590 flowers per vine and a normalized root mean squared error (NRMSE) of 23.7% was obtained. An R2 above 0.70 was achieved between the estimated number of actual flowers and the final yield, per vine. These results show that the number of flowers per vine can be estimated using machine vision and deep learning. The developed imaging platform can be used by the wine industry in commercial vineyards for a satisfactory early crop yield forecasting.