Global Wheat Challenge 2020: Analysis of the competition design and winning models

1. Abstract Data competitions have become a popular approach to crowdsource new data analysis methods for general and specialized data science problems. In plant phenotyping, data competitions have a rich history, and new outdoor field datasets have potential for new data competitions. We developed the Global Wheat Challenge as a generalization competition to see if solutions for wheat head detection from field images would work in different regions around the world. In this paper, we analyze the winning challenge solutions in terms of their robustness and the relative importance of model and data augmentation design decisions. We found that the design of the competition influence the selection of winning solutions and provide recommendations for future competitions in an attempt to garner more robust winning solutions.

[1]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[2]  Hervé Goëau,et al.  Overview of LifeCLEF Plant Identification task 2020 , 2020, CLEF.

[3]  Anders Krogh Mortensen,et al.  The GrassClover Image Dataset for Semantic and Hierarchical Species Understanding in Agriculture , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[4]  Julio Saez-Rodriguez,et al.  Crowdsourcing Network Inference: The DREAM Predictive Signaling Network Challenge , 2011, Science Signaling.

[5]  David P. Anderson,et al.  Status of the UC-Berkeley SETI efforts , 2011, Optical Engineering + Applications.

[6]  F. Baret,et al.  Global Wheat Head Detection (GWHD) Dataset: A Large and Diverse Dataset of High-Resolution RGB-Labelled Images to Develop and Benchmark Wheat Head Detection Methods , 2020, Plant phenomics.

[7]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[8]  Hanno Scharr,et al.  Leaf segmentation in plant phenotyping: a collation study , 2016, Machine Vision and Applications.

[9]  Seong Joon Oh,et al.  CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[11]  J. Dunning.  The elephant in the room. , 2013, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[12]  Aaron Bauer,et al.  De novo protein design by citizen scientists , 2019, Nature.

[13]  Huiji Gao,et al.  Harnessing the Crowdsourcing Power of Social Media for Disaster Relief , 2011, IEEE Intelligent Systems.

[14]  Li Wang,et al.  Deep Learning for Practical Image Recognition: Case Study on Kaggle Competitions , 2018, KDD.

[15]  R. Solovyev,et al.  Weighted Boxes Fusion: ensembling boxes for object detection models , 2019, ArXiv.

[16]  Hao Lu,et al.  TasselNetv2: in-field counting of wheat spikes with context-augmented local regression networks , 2019, Plant Methods.

[17]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[18]  Percy Liang,et al.  Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization , 2019, ArXiv.

[19]  Jure Leskovec,et al.  WILDS: A Benchmark of in-the-Wild Distribution Shifts , 2021, ICML.

[20]  Hanno Scharr,et al.  Finely-grained annotated datasets for image-based plant phenotyping , 2016, Pattern Recognit. Lett..

[21]  Kai Zhang,et al.  The Plant Pathology Challenge 2020 data set to classify foliar disease of apples , 2020, Applications in plant sciences.

[22]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[23]  Quoc V. Le,et al.  EfficientDet: Scalable and Efficient Object Detection , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Yunchao Wei,et al.  Agriculture-Vision: A Large Aerial Image Database for Agricultural Pattern Analysis , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Rasmus Nyholm Jørgensen,et al.  A Public Image Database for Benchmark of Plant Seedling Classification Algorithms , 2017, ArXiv.

[26]  M. Bethge,et al.  Shortcut learning in deep neural networks , 2020, Nature Machine Intelligence.

[27]  Hanno Scharr,et al.  Citizen crowds and experts: observer variability in image-based plant phenotyping , 2018, Plant Methods.

[28]  Kevin Crowston,et al.  From Conservation to Crowdsourcing: A Typology of Citizen Science , 2011, 2011 44th Hawaii International Conference on System Sciences.

[29]  Hanno Scharr,et al.  Sharing the Right Data Right: A Symbiosis with Machine Learning. , 2019, Trends in plant science.

[30]  Pierre Bonnet,et al.  Overview of LifeCLEF Plant Identification Task 2019: diving into Data Deficient Tropical Countries , 2019, CLEF.

[31]  Plant detection and counting from high-resolution RGB images acquired from UAVs: comparison between deep-learning and handcrafted methods with application to maize, sugar beet, and sunflower crops , 2021, bioRxiv.

[32]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction , 2018, ArXiv.

[33]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[34]  Frédéric Baret,et al.  Ear density estimation from high resolution RGB imagery using deep learning technique , 2019, Agricultural and Forest Meteorology.