Tomato leaf segmentation algorithms for mobile phone applications using deep learning

Abstract Leaf disease recognition using image processing techniques is currently a hot research topic. High disease recognition accuracies have been reported in literature particularly in studies that relied on lab images captured under controlled lighting and uniform background conditions. When these systems were tested under real field conditions, however, their performance dropped sharply. Studies have demonstrated that complex backgrounds significantly contribute to this drop in performance and that background removal enhances disease recognition accuracy. Unfortunately, a fast and accurate means to perform automatic background subtraction for leaf images has so far not yet been developed. In this paper, we propose fully convolutional neural networks to perform automatic background subtraction for leaf images captured in mobile applications. In a mobile application use case, the target leaf would typically dominate an image captured by the farmer. The leaf would also be surrounded by various background features including other leaves, stems, fruits, soil and mulch. The goal of the segmentation network is to remove these background features so that only the target leaf remains. A dataset that is representative of this scenario was prepared in order to train and test the proposed networks. It consists of 1,408 tomato leaf images captured under challenging field conditions and their respective ground truth masks. We report state-of-the-art leaf image segmentation performance of over 0.96 mean weighted intersection over union and over 0.91 boundary F1 score. In particular, our proposed segmentation network KijaniNet outperforms all competitors scoring 0.9766 mean weighted intersection over union and 0.9439 boundary F1 score. The proposed technique supersedes competing background subtraction algorithms yet does not require user intervention nor does it place constraints on the orientation, shape or illumination of the target leaf. Furthermore, all CNN models are able to perform segmentation of a 256x256 pixel RGB image in under 0.12 s when running on a GPU and in less than 2.1 s when running on a CPU; which is much faster than any of the competing techniques.

[1]  Fuji Ren,et al.  Feature Reuse Residual Networks for Insect Pest Recognition , 2019, IEEE Access.

[2]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[3]  Ricardo Matsumura de Araújo,et al.  Classification of Apple Tree Disorders Using Convolutional Neural Networks , 2016, 2016 IEEE 28th International Conference on Tools with Artificial Intelligence (ICTAI).

[4]  A. K. Misra,et al.  Detection of plant leaf diseases using image segmentation and soft computing techniques , 2017 .

[5]  David Zhang,et al.  A survey of graph theoretical approaches to image segmentation , 2013, Pattern Recognit..

[6]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  James Patrick Underwood,et al.  Deep fruit detection in orchards , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[8]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[9]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Uday Pratap Singh,et al.  Bacterial Foraging Optimization Based Radial Basis Function Neural Network (BRBFNN) for Identification and Classification of Plant Leaf Diseases: An Automatic Approach Towards Plant Pathology , 2018, IEEE Access.

[11]  Bülent Sankur,et al.  Survey over image thresholding techniques and quantitative performance evaluation , 2004, J. Electronic Imaging.

[12]  Xin Zhao,et al.  Use of Unmanned Aerial Vehicle Imagery and Deep Learning UNet to Extract Rice Lodging , 2019, Sensors.

[13]  Gabriela Csurka,et al.  What is a good evaluation measure for semantic segmentation? , 2013, BMVC.

[14]  Florentin Smarandache,et al.  Smart mobile application to recognize tomato leaf diseases using Convolutional Neural Networks , 2019, 2019 International Conference of Computer Science and Renewable Energies (ICCSRE).

[15]  Chit Su Hlaing,et al.  Tomato Plant Diseases Classification Using Statistical Texture Feature and Color Feature , 2018, 2018 IEEE/ACIS 17th International Conference on Computer and Information Science (ICIS).

[16]  Torsten Bertram,et al.  A Fast Multi-Task CNN for Spatial Understanding of Traffic Scenes , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[17]  Dong Sun Park,et al.  High-Performance Deep Neural Network-Based Tomato Plant Diseases and Pests Diagnosis System With Refinement Filter Bank , 2018, Front. Plant Sci..

[18]  Xiaofeng Wang,et al.  Classification of plant leaf images with complicated background , 2008, Appl. Math. Comput..

[19]  Artzai Picón,et al.  Automatic plant disease diagnosis using mobile capture devices, applied on a wheat use case , 2017, Comput. Electron. Agric..

[20]  Konstantinos P. Ferentinos,et al.  Deep learning models for plant disease detection and diagnosis , 2018, Comput. Electron. Agric..

[21]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Yun Zhang,et al.  Identification of Apple Leaf Diseases Based on Deep Convolutional Neural Networks , 2017, Symmetry.

[23]  Sang Cheol Kim,et al.  A Robust Deep-Learning-Based Detector for Real-Time Tomato Plant Diseases and Pests Recognition , 2017, Sensors.

[24]  Abdelouahab Moussaoui,et al.  Deep Learning for Tomato Diseases: Classification and Symptoms Visualization , 2017, Appl. Artif. Intell..

[25]  Serge Beucher,et al.  Use of watersheds in contour detection , 1979 .

[26]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[27]  Jayme Garcia Arnal Barbedo,et al.  Plant disease identification from individual lesions and spots using deep learning , 2019, Biosystems Engineering.

[28]  Henry Medeiros,et al.  Multispecies Fruit Flower Detection Using a Refined Semantic Segmentation Network , 2018, IEEE Robotics and Automation Letters.

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  J. Yosinski,et al.  Automated Identification of Northern Leaf Blight-Infected Maize Plants from Field Imagery Using Deep Learning. , 2017, Phytopathology.

[31]  Darko Stefanovic,et al.  Deep Neural Networks Based Recognition of Plant Diseases by Leaf Image Classification , 2016, Comput. Intell. Neurosci..

[32]  Alsayed Algergawy,et al.  A Deep Learning-based Approach for Banana Leaf Diseases Classification , 2017, BTW.

[33]  Kushtrim Bresilla,et al.  Single-Shot Convolution Neural Networks for Real-Time Fruit Detection Within the Tree , 2019, Front. Plant Sci..

[34]  Andrea Luvisi,et al.  X-FIDO: An Effective Application for Detecting Olive Quick Decline Syndrome with Deep Learning and Data Fusion , 2017, Front. Plant Sci..

[35]  Ghulam Muhammad,et al.  Automatic Fruit Classification Using Deep Learning for Industrial Applications , 2019, IEEE Transactions on Industrial Informatics.

[36]  Jayme Garcia Arnal Barbedo,et al.  A review on the main challenges in automatic plant disease identification based on visible range images , 2016 .

[37]  Jayme Garcia Arnal Barbedo,et al.  Factors influencing the use of deep learning for plant disease recognition , 2018, Biosystems Engineering.

[38]  Marcel Salathé,et al.  Using Deep Learning for Image-Based Plant Disease Detection , 2016, Front. Plant Sci..

[39]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[40]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Zhu-Hong You,et al.  Plant disease leaf image segmentation based on superpixel clustering and EM algorithm , 2017, Neural Computing and Applications.

[42]  Yang Lu,et al.  Identification of rice diseases using deep convolutional neural networks , 2017, Neurocomputing.

[43]  Artzai Picón,et al.  Deep convolutional neural networks for mobile capture device-based crop disease classification in the wild , 2019, Comput. Electron. Agric..

[44]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[45]  Darko Stefanovic,et al.  Solving Current Limitations of Deep Learning Based Approaches for Plant Disease Detection , 2019, Symmetry.

[46]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[47]  Kemal Adem,et al.  Automatic detection and classification of leaf spot disease in sugar beet using deep learning algorithms , 2019, Physica A: Statistical Mechanics and its Applications.

[48]  Tim Cootes,et al.  An Introduction to Active Shape Models , 2000 .

[49]  Hitoshi Iyatomi,et al.  Basic Investigation on a Robust and Practical Plant Diagnostic System , 2016, 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA).

[50]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[51]  Jayaram K. Udupa,et al.  An ultra-fast user-steered image segmentation paradigm: live wire on the fly , 2000, IEEE Transactions on Medical Imaging.

[52]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).