Precision Detection and Assessment of Ash Death and Decline Caused by the Emerald Ash Borer Using Drones and Deep Learning

Emerald ash borer (Agrilus planipennis) is an invasive pest that has killed millions of ash trees (Fraxinus spp.) in the USA since its first detection in 2002. Although the current methods for trapping emerald ash borers (e.g., sticky traps and trap trees) and visual ground and aerial surveys are generally effective, they are inefficient for precisely locating and assessing the declining and dead ash trees in large or hard-to-access areas. This study was conducted to develop and evaluate a new tool for safe, efficient, and precise detection and assessment of ash decline and death caused by emerald ash borer by using aerial surveys with unmanned aerial systems (a.k.a., drones) and a deep learning model. Aerial surveys with drones were conducted to obtain 6174 aerial images including ash decline in the deciduous forests in West Virginia and Pennsylvania, USA. The ash trees in each image were manually annotated for training and validating deep learning models. The models were evaluated using the object recognition metrics: mean average precisions (mAP) and two average precisions (AP50 and AP75). Our comprehensive analyses with instance segmentation models showed that Mask2former was the most effective model for detecting declining and dead ash trees with 0.789, 0.617, and 0.542 for AP50, AP75, and mAP, respectively, on the validation dataset. A follow-up in-situ field study conducted in nine locations with various levels of ash decline and death demonstrated that deep learning along with aerial survey using drones could be an innovative tool for rapid, safe, and efficient detection and assessment of ash decline and death in large or hard-to-access areas.

[1]  D. Goodin,et al.  Leaf-Level Spectroscopy for Analysis of Invasive Pest Impact on Trees in a Stressed Environment: An Example Using Emerald Ash Borer (Agrilus planipennis Fairmaire) in Ash Trees (Fraxinus spp.), Kansas, USA , 2022, Environments.

[2]  A. Schwing,et al.  Masked-attention Mask Transformer for Universal Image Segmentation , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Russell G. Congalton,et al.  Monitoring Fine-Scale Forest Health Using Unmanned Aerial Systems (UAS) Multispectral Models , 2021, Remote. Sens..

[4]  Alexander G. Schwing,et al.  Per-Pixel Classification is Not All You Need for Semantic Segmentation , 2021, NeurIPS.

[5]  Xianzhi Du,et al.  Simple Training Strategies and Model Scaling for Object Detection , 2021, ArXiv.

[6]  Gwan-Seok Lee,et al.  Detection of Monema flavescens (Lepidoptera: Limacodidae) Cocoons Using Small Unmanned Aircraft System , 2021, Journal of Economic Entomology.

[7]  Chang-Gyu Park,et al.  Advances, Limitations, and Future Applications of Aerospace and Geospatial Technologies for Apple IPM , 2021 .

[8]  Matthieu Cord,et al.  Training data-efficient image transformers & distillation through attention , 2020, ICML.

[9]  Jifeng Ning,et al.  Identifying sunflower lodging based on image fusion and deep semantic segmentation with UAV remote sensing imaging , 2020, Comput. Electron. Agric..

[10]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[11]  D. Coomes,et al.  Monitoring ash dieback (Hymenoscyphus fraxineus) in British forests using hyperspectral remote sensing , 2020, Remote Sensing in Ecology and Conservation.

[12]  Bishwa B. Sapkota,et al.  High-resolution mapping of ash (Fraxinus spp.) in bottomland hardwoods to slow Emerald Ash Borer infestation , 2020, Science of Remote Sensing.

[13]  Nicolas Usunier,et al.  End-to-End Object Detection with Transformers , 2020, ECCV.

[14]  Ming-Der Yang,et al.  Semantic Segmentation Using Deep Learning with Vegetation Indices for Rice Lodging Identification in Multi-date UAV Visible Images , 2020, Remote. Sens..

[15]  Wei Chen Tseng,et al.  Real-time Crop Classification Using Edge Computing and Deep Learning , 2020, 2020 IEEE 17th Annual Consumer Communications & Networking Conference (CCNC).

[16]  Taghi M. Khoshgoftaar,et al.  A survey on Image Data Augmentation for Deep Learning , 2019, Journal of Big Data.

[17]  Kai Chen,et al.  MMDetection: Open MMLab Detection Toolbox and Benchmark , 2019, ArXiv.

[18]  Yong Jae Lee,et al.  YOLACT: Real-Time Instance Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Alex Sherstinsky,et al.  Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network , 2018, Physica D: Nonlinear Phenomena.

[20]  Joseph Redmon,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[21]  Carsten Rother,et al.  Panoptic Segmentation , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Kaiming He,et al.  Focal Loss for Dense Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[24]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[25]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[27]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Jian Yang,et al.  Ash Decline Assessment in Emerald Ash Borer Infested Natural Forests Using High Spatial Resolution Images , 2016, Remote. Sens..

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[31]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[34]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[36]  LeRoy Rodgers,et al.  Mapping Invasive Plant Distributions in the Florida Everglades Using the Digital Aerial Sketch Mapping Technique , 2014, Invasive Plant Science and Management.

[37]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[38]  D. Herms,et al.  Emerald ash borer invasion of North America: history, biology, ecology, impacts, and management. , 2014, Annual review of entomology.

[39]  C. Stone,et al.  Aerial mapping canopy damage by the aphid Essigella californica in a Pinus radiata plantation in southern New South Wales: what are the challenges? , 2013 .

[40]  Derek Hoiem,et al.  Diagnosing Error in Object Detectors , 2012, ECCV.

[41]  D. Maclean,et al.  Validation of Spruce Budworm Outbreak History Developed from Aerial Sketch Mapping of Defoliation in New Brunswick , 2008 .

[42]  Rayda K. Krell,et al.  Theory, Technology, and Practice of Site-Specific Insect Pest Management , 2007 .

[43]  Elizabeth A. Boyd,et al.  Mechanical and Insect Transmission of Xylella fastidiosa to Vitis vinifera , 2007, American Journal of Enology and Viticulture.

[44]  Therese M. Poland,et al.  Emerald ash borer in North America: a research and regulatory challenge , 2005 .

[45]  Stephen Lin,et al.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[46]  Р Ю Чуйков,et al.  Обнаружение транспортных средств на изображениях загородных шоссе на основе метода Single shot multibox Detector , 2017 .

[47]  T. Poland,et al.  The Emerald Ash Borer: A New Exotic Pest in North America , 2002 .