Data-centric annotation analysis for plant disease detection: Strategy, consistency, and performance

Object detection models have become the current tool of choice for plant disease detection in precision agriculture. Most existing research improved the performance by ameliorating networks and optimizing the loss function. However, because of the vast influence of data annotation quality and the cost of annotation, the data-centric part of a project also needs more investigation. We should further consider the relationship between data annotation strategies, annotation quality, and the model’s performance. In this paper, a systematic strategy with four annotation strategies for plant disease detection is proposed: local, semi-global, global, and symptom-adaptive annotation. Labels with different annotation strategies will result in distinct models’ performance, and their contrasts are remarkable. An interpretability study of the annotation strategy is conducted by using class activation maps. In addition, we define five types of inconsistencies in the annotation process and investigate the severity of the impact of inconsistent labels on model’s performance. Finally, we discuss the problem of label inconsistency during data augmentation. Overall, this data-centric quantitative analysis helps us to understand the significance of annotation strategies, which provides practitioners a way to obtain higher performance and reduce annotation costs on plant disease detection. Our work encourages researchers to pay more attention to annotation consistency and the essential issues of annotation strategy. The code will be released at: https://github.com/JiuqingDong/PlantDiseaseDetection_Yolov5 .

[1]  D. Park,et al.  A Comprehensive Survey of Image Augmentation Techniques for Deep Learning , 2022, Pattern Recognit..

[2]  Ziming Zhang,et al.  Robust Object Detection With Inaccurate Bounding Boxes , 2022, ECCV.

[3]  A. Valros,et al.  Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation , 2022, NeurIPS.

[4]  Shengshan Hu,et al.  Towards Efficient Data-Centric Robust Machine Learning with Noise-based Augmentation , 2022, ArXiv.

[5]  Li Dong,et al.  Swin Transformer V2: Scaling Up Capacity and Resolution , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Vinodkumar Prabhakaran,et al.  Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations , 2021, TACL.

[7]  Mariusz Rafało,et al.  Cross validation methods: Analysis based on diagnostics of thyroid cancer metastasis , 2021, ICT Express.

[8]  Jucheng Yang,et al.  Style-Consistent Image Translation: A Novel Data Augmentation Paradigm to Improve Plant Disease Recognition , 2022, Frontiers in Plant Science.

[9]  Barbara Plank,et al.  Learning from Disagreement: A Survey , 2021, J. Artif. Intell. Res..

[10]  Xuewei Chao,et al.  Toward Sustainability: Trade-Off Between Data Quality and Quantity in Crop Pest Recognition , 2021, Frontiers in Plant Science.

[11]  D. Park,et al.  Open Set Self and Across Domain Adaptation for Tomato Disease Recognition With Deep Learning Techniques , 2021, Frontiers in Plant Science.

[12]  Matthias Schubert,et al.  Correcting Imprecise Object Locations for Training Object Detectors in Remote Sensing Applications , 2021, Remote. Sens..

[13]  Francesca Maridina Malloci,et al.  DiaMOS Plant: A Dataset for Diagnosis and Monitoring Plant Disease , 2021, Agronomy.

[14]  Yagya Raj Pandeya,et al.  An Instance Segmentation Model for Strawberry Diseases Based on Mask R-CNN , 2021, Sensors.

[15]  Ming-Ming Cheng,et al.  LayerCAM: Exploring Hierarchical Class Activation Maps for Localization , 2021, IEEE Transactions on Image Processing.

[16]  Fei Wu,et al.  Training Robust Object Detectors From Noisy Category Labels and Imprecise Bounding Boxes , 2021, IEEE Transactions on Image Processing.

[17]  Quoc V. Le,et al.  EfficientNetV2: Smaller Models and Faster Training , 2021, ICML.

[18]  L. Emmi,et al.  A hybrid representation of the environment to improve autonomous navigation of mobile robots in agriculture , 2021 .

[19]  G. Algan,et al.  Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey , 2019, Knowl. Based Syst..

[20]  Valerio Basile,et al.  We Need to Consider Disagreement in Evaluation , 2021, BPPF.

[21]  Kiyoharu Aizawa,et al.  Noisy Localization Annotation Refinement For Object Detection , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[22]  Bernhard Schauberger,et al.  A systematic review of local to regional yield forecasting approaches and frequently used data resources , 2020 .

[23]  Xin Zhang,et al.  Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN , 2020, Comput. Electron. Agric..

[24]  Krishan Kumar,et al.  Plant disease detection using computational intelligence and image processing , 2020, Journal of Plant Diseases and Protection.

[25]  Baohua Zhang,et al.  Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots , 2020, Comput. Electron. Agric..

[26]  Mohammed Bany Muhammad,et al.  Eigen-CAM: Class Activation Map using Principal Components , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[27]  Sylvia Y. He,et al.  Pandemics of People and Plants: Which Is the Greater Threat to Food Security? , 2020, Molecular Plant.

[28]  Junnan Li,et al.  Towards Noise-resistant Object Detection with Noisy Annotations , 2020, ArXiv.

[29]  Jianhui Lin,et al.  Using Deep Learning for Image-Based Different Degrees of Ginkgo Leaf Disease Classification , 2020, Inf..

[30]  Naman Jain,et al.  PlantDoc: A Dataset for Visual Plant Disease Detection , 2019, COMAD/CODS.

[31]  Fan Yang,et al.  Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Dong Sun Park,et al.  Unsupervised Image Translation using Adversarial Networks for Improved Plant Disease Recognition , 2019, Comput. Electron. Agric..

[34]  Johan Potgieter,et al.  Plant Disease Detection and Classification by Deep Learning , 2019, Plants.

[35]  Jorge Parraga-Alava,et al.  RoCoLe: A robusta coffee leaf images dataset for evaluation of machine learning based methods in plant diseases recognition , 2019, Data in brief.

[36]  Quoc V. Le,et al.  AutoAugment: Learning Augmentation Strategies From Data , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Quoc V. Le,et al.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks , 2019, ICML.

[38]  Jayme Garcia Arnal Barbedo,et al.  Plant disease identification from individual lesions and spots using deep learning , 2019, Biosystems Engineering.

[39]  Neil McRoberts,et al.  The global burden of pathogens and pests on major food crops , 2019, Nature Ecology & Evolution.

[40]  Jakub Nalepa,et al.  Validating Hyperspectral Image Segmentation , 2018, IEEE Geoscience and Remote Sensing Letters.

[41]  Bernard Ghanem,et al.  Missing Labels in Object Detection , 2019, CVPR Workshops.

[42]  Dong Sun Park,et al.  High-Performance Deep Neural Network-Based Tomato Plant Diseases and Pests Diagnosis System With Refinement Filter Bank , 2018, Front. Plant Sci..

[43]  Hod Lipson,et al.  Image set for deep learning: field images of maize annotated with disease symptoms , 2018, BMC Research Notes.

[44]  Kenichi Kobayashi,et al.  Evaluation of Data Augmentation for Image-Based Plant-Disease Detection , 2018, 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[45]  Konstantinos P. Ferentinos,et al.  Deep learning models for plant disease detection and diagnosis , 2018, Comput. Electron. Agric..

[46]  Yun Zhang,et al.  Identification of Apple Leaf Diseases Based on Deep Convolutional Neural Networks , 2017, Symmetry.

[47]  Dongshik Kang,et al.  Evaluation of Class Noise Impact on Performance of Machine Learning Algorithms , 2018 .

[48]  Sang Cheol Kim,et al.  A Robust Deep-Learning-Based Detector for Real-Time Tomato Plant Diseases and Pests Recognition , 2017, Sensors.

[49]  R. Goodhue,et al.  Crop Disease and Agricultural Productivity , 2017 .

[50]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[51]  Marcel Salathé,et al.  An open access repository of images on plant health to enable the development of mobile disease diagnostics through machine learning and crowdsourcing , 2015, ArXiv.

[52]  David Flatow Stanford.  On the Robustness of ConvNets to Training on Noisy Labels , 2015 .

[53]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[54]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[55]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Xingquan Zhu,et al.  Class Noise vs. Attribute Noise: A Quantitative Study , 2003, Artificial Intelligence Review.