Comparison of Object Detection and Patch-Based Classification Deep Learning Models on Mid- to Late-Season Weed Detection in UAV Imagery

Midto late-season weeds that escape from the routine early-season weed management threaten agricultural production by creating a large number of seeds for several future growing seasons. Rapid and accurate detection of weed patches in field is the first step of site-specific weed management. In this study, object detection-based convolutional neural network models were trained and evaluated over low-altitude unmanned aerial vehicle (UAV) imagery for midto late-season weed detection in soybean fields. The performance of two object detection models, Faster RCNN and the Single Shot Detector (SSD), were evaluated and compared in terms of weed detection performance using mean Intersection over Union (IoU) and inference speed. It was found that the Faster RCNN model with 200 box proposals had similar good weed detection performance to the SSD model in terms of precision, recall, f1 score, and IoU, as well as a similar inference time. The precision, recall, f1 score and IoU were 0.65, 0.68, 0.66 and 0.85 for Faster RCNN with 200 proposals, and 0.66, 0.68, 0.67 and 0.84 for SSD, respectively. However, the optimal confidence threshold of the SSD model was found to be much lower than that of the Faster RCNN model, which indicated that SSD might have lower generalization performance than Faster RCNN for midto late-season weed detection in soybean fields using UAV imagery. The performance of the object detection model was also compared with patch-based CNN model. The Faster RCNN model yielded a better weed detection performance than the patch-based CNN with and without overlap. The inference time of Faster RCNN was similar to patch-based CNN without overlap, but significantly less than patch-based CNN with overlap. Hence, Faster RCNN was found to be the best model in terms of weed detection performance and inference time among the different models compared in this study. This work is important in understanding the potential and identifying the algorithms for an on-farm, near real-time weed detection and management.

[1]  Córdova-Cruzatty Andrea,et al.  Precise weed and maize classification through convolutional neuronal networks , 2017, 2017 IEEE Second Ecuador Technical Chapters Meeting (ETCM).

[2]  Joachim Denzler,et al.  Active Learning for Deep Object Detection , 2018, VISIGRAPP.

[3]  Cyrill Stachniss,et al.  UAV-based crop and weed classification for smart farming , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Alexei A. Efros,et al.  Unsupervised Visual Representation Learning by Context Prediction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  F. López-Granados,et al.  Configuration and Specifications of an Unmanned Aerial Vehicle (UAV) for Early Site Specific Weed Management , 2013, PloS one.

[6]  Gavriel Salomon,et al.  T RANSFER OF LEARNING , 1992 .

[7]  J. Nielsen,et al.  Pre-harvest weed mapping of Cirsium arvense in wheat and barley with off-the-shelf UAVs , 2018, Precision Agriculture.

[8]  Arnold W. Schumann,et al.  Weed Detection in Perennial Ryegrass With Deep Learning Convolutional Neural Network , 2019, Front. Plant Sci..

[9]  Jizhong Deng,et al.  A fully convolutional network for weed mapping of unmanned aerial vehicle (UAV) imagery , 2018, PloS one.

[10]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[11]  Francisca López-Granados,et al.  Spectral discrimination of Ridolfia segetum and sunflower as affected by phenological stage , 2006 .

[12]  Francisca López-Granados,et al.  Object- and pixel-based analysis for mapping crops and their agro-environmental associated measures using QuickBird imagery , 2009 .

[13]  Cyrill Stachniss,et al.  Fully Convolutional Networks With Sequential Information for Robust Crop and Weed Detection in Precision Farming , 2018, IEEE Robotics and Automation Letters.

[14]  Wen Zhang,et al.  A review on weed detection using ground-based machine vision and image processing techniques , 2019, Comput. Electron. Agric..

[15]  Weihong Deng,et al.  Very deep convolutional neural network based image classification using small training sample size , 2015, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR).

[16]  M. F. Kocher,et al.  Textural imaging and discriminant analysis for distinguishing weeds for spot spraying , 1998 .

[17]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[18]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Jorge Torres-Sánchez,et al.  An Automatic Random Forest-OBIA Algorithm for Early Weed Mapping between and within Crop Rows Using UAV Imagery , 2018, Remote. Sens..

[20]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[21]  L. M. Navas-Gracia,et al.  UAV Detection of Sinapis arvensis Infestation in Alfalfa Plots Using Simple Vegetation Indices from Conventional Digital Cameras , 2020, AgriEngineering.

[22]  T. F. Burks,et al.  CLASSIFICATION OF WEED SPECIES USING COLOR TEXTURE FEATURES AND DISCRIMINANT ANALYSIS , 2000 .

[23]  Roland Siegwart,et al.  weedNet: Dense Semantic Weed Classification Using Multispectral Images and MAV for Smart Farming , 2017, IEEE Robotics and Automation Letters.

[24]  Cyrill Stachniss,et al.  WeedMap: A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming , 2018, Remote. Sens..

[25]  F. López-Granados,et al.  Multi-temporal mapping of the vegetation fraction in early-season wheat fields using images from UAV , 2014 .

[26]  C. H. Koger,et al.  Detecting Late-Season Weed Infestations in Soybean (Glycine max)1 , 2003, Weed Technology.

[27]  P. Dutilleul,et al.  CLASSIFICATION ACCURACY OF DISCRIMINANT ANALYSIS, ARTIFICIAL NEURAL NETWORKS, AND DECISION TREES FOR WEED AND NITROGEN STRESS DETECTION IN CORN , 2005 .

[28]  Francisca López-Granados,et al.  Applying Neural Networks to Hyperspectral and Multispectral Field Data for Discrimination of Cruciferous Weeds in Winter Crops , 2012, TheScientificWorldJournal.

[29]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[30]  F. López-Granados,et al.  Weed Mapping in Early-Season Maize Fields Using Object-Based Analysis of Unmanned Aerial Vehicle (UAV) Images , 2013, PloS one.

[31]  Hemerson Pistori,et al.  Weed detection in soybean crops using ConvNets , 2017, Comput. Electron. Agric..

[32]  Francisca López-Granados,et al.  Broad-scale cruciferous weed patch classification in winter wheat using QuickBird imagery for in-season site-specific control , 2013, Precision Agriculture.

[33]  L. Tian,et al.  A Review on Remote Sensing of Weeds in Agriculture , 2004, Precision Agriculture.

[34]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[35]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  F. López-Granados,et al.  Early season weed mapping in sunflower using UAV technology: variability of herbicide treatment maps against weed thresholds , 2016, Precision Agriculture.

[37]  R. Gerhards,et al.  Precision farming for weed management: techniques , 2008, Gesunde Pflanzen.

[38]  Francisca López-Granados,et al.  Weed detection for site-specific weed management: mapping and real-time approaches , 2011 .

[39]  J. O'donovan,et al.  Influence of the Relative Time of Emergence of Wild Oat (Avena fatua) on Yield Loss of Barley (Hordeum vulgare) and Wheat (Triticum aestivum) , 1985, Weed Science.

[40]  Esmaeil S. Nadimi,et al.  Site‐specific weed control technologies , 2009 .

[41]  F. Castaldi,et al.  Assessing the potential of images from unmanned aerial vehicles (UAV) to support herbicide patch spraying in maize , 2017, Precision Agriculture.

[42]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[43]  김용한,et al.  Single Shot Multibox Detector를 적용한 실시간 선박 검출 및 분류 시스템 개발 , 2020 .

[44]  Francisca López Granados.  Weed detection for site-specific weed management: Mapping and real-time approaches , 2011 .

[45]  Bhagirath S. Chauhan,et al.  Ecology and management of weeds under conservation agriculture: A review , 2012 .

[46]  The Santaló-regions of a convex body , 1997, math/9707209.

[47]  J. Neal,et al.  Response of Japanese Stiltgrass (Microstegium vimineum) to Application Timing, Rate, and Frequency of Postemergence Herbicides1 , 2005, Weed Technology.

[48]  Francisca López-Granados,et al.  Airborne multi-spectral imagery for mapping cruciferous weeds in cereal and legume crops , 2011, Precision Agriculture.

[49]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[50]  Clarence J. Swanton,et al.  Integrated Weed Management: Knowledge-Based Weed Management Systems , 2008, Weed Science.

[51]  Cyrill Stachniss,et al.  Real-Time Semantic Segmentation of Crop and Weed for Precision Agriculture Robots Leveraging Background Knowledge in CNNs , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[52]  Adel Hafiane,et al.  Deep Learning Based Classification System for Identifying Weeds Using High-Resolution UAV Imagery , 2018, Advances in Intelligent Systems and Computing.

[53]  Patrick D. Gerard,et al.  Utility of Multispectral Imagery for Soybean and Weed Species Differentiation , 2008, Weed Technology.

[54]  Ryosuke Shibasaki,et al.  Estimating crop yields with deep learning and remotely sensed data , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[55]  Pedro Antonio Gutiérrez,et al.  A semi-supervised system for weed mapping in sunflower crops using unmanned aerial vehicles and a crop row detection method , 2015, Appl. Soft Comput..

[56]  F. López-Granados,et al.  Is the current state of the art of weed monitoring suitable for site-specific weed management in arable crops? , 2018 .

[57]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  David Riaño,et al.  Assessing the potential of hyperspectral remote sensing for the discrimination of grassweeds in winter cereal crops , 2011 .

[60]  Cyrill Stachniss,et al.  Robust joint stem detection and crop‐weed classification using image sequences for plant‐specific treatment in precision farming , 2019, J. Field Robotics.

[61]  Sergio Guadarrama,et al.  Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Fan Xia,et al.  Assessing object-based classification: advantages and limitations , 2009 .

[63]  Peter Christiansen,et al.  Using Deep Learning to Challenge Safety Standard for Highly Autonomous Machines in Agriculture , 2016, J. Imaging.

[64]  Marcel Salathé,et al.  Using Deep Learning for Image-Based Plant Disease Detection , 2016, Front. Plant Sci..

[65]  S. Sankaran,et al.  Low-altitude, high-resolution aerial imaging systems for row and field crop phenotyping: A review , 2015 .

[66]  César Fernández-Quintanilla,et al.  Spatial stability of Avena sterilis ssp. ludoviciana populations under annual applications of low rates of imazamethabenz , 2004 .

[67]  Feng Liu,et al.  Modeling spatio-temporal distribution of soil moisture by deep learning-based cellular automata model , 2016, Journal of Arid Land.

[68]  Anders Krogh Mortensen,et al.  Pixel-wise classification of weeds and crops in images by using a Fully Convolutional neural network , 2016 .

[69]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[70]  N. Zhang,et al.  Precision agriculture—a worldwide overview , 2002 .

[71]  Stefano Pignatti,et al.  UAV-based hyperspectral imaging for weed discrimination in maize , 2019, Precision agriculture ’19.