Detecting and mapping tree seedlings in UAV imagery using convolutional neural networks and field-verified data

Abstract Mapping of tree seedlings is useful for tasks ranging from monitoring natural succession and regeneration to effective silvicultural management. Development of methods that are both accurate and cost-effective is especially important considering the dramatic increase in tree planting that is required globally to mitigate the impacts of climate change. The combination of high-resolution imagery from unmanned aerial vehicles and object detection by convolutional neural networks (CNNs) is one promising approach. However, unbiased assessments of these models and methods to integrate them into geospatial workflows are lacking. In this study, we present a method for rapid, large-scale mapping of young conifer seedlings using CNNs applied to RGB orthomosaic imagery. Importantly, we provide an unbiased assessment of model performance by using two well-characterised trial sites together containing over 30,000 seedlings to assemble datasets with a high level of completeness. Our results showed CNN-based models trained on two sites detected seedlings with sensitivities of 99.5% and 98.8%. False positives due to tall weeds at one site and naturally regenerating seedlings of the same species led to slightly lower precision of 98.5% and 96.7%. A model trained on examples from both sites had 99.4% sensitivity and precision of 97%, showing applicability across sites. Additional testing showed that the CNN model was able to detect 68.7% of obscured seedlings missed during the initial annotation of the imagery but present in the field data. Finally, we demonstrate the potential to use a form of weakly supervised training and a tile-based processing chain to enhance the accuracy and efficiency of CNNs applied to large, high-resolution orthomosaics.

[1]  Nguyen Duy Hung,et al.  Deep learning based banana plant detection and counting using high-resolution red-green-blue (RGB) images collected from unmanned aerial vehicle (UAV) , 2019, PloS one.

[2]  Jonathan P. Dash,et al.  Taking a closer look at invasive alien plant research: A review of the current state, opportunities, and future directions for UAVs , 2019, Methods in Ecology and Evolution.

[3]  Joanne C. White,et al.  Remote Sensing Technologies for Enhancing Forest Inventories: A Review , 2016 .

[4]  S. Popescu,et al.  Measuring individual tree crown diameter with lidar and assessing its influence on estimating forest volume and biomass , 2003 .

[5]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[6]  Vincent G. Ambrosia,et al.  Unmanned Aircraft Systems in Remote Sensing and Scientific Research: Classification and Considerations of Use , 2012, Remote. Sens..

[7]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[8]  Juha Hyyppä,et al.  An International Comparison of Individual Tree Detection and Extraction Using Airborne Laser Scanning , 2012, Remote. Sens..

[9]  Peter Z. Fulé,et al.  Monitoring Landscape‐Scale Ponderosa Pine Restoration Treatment Implementation and Effectiveness , 2010 .

[10]  Lisa M. Brown,et al.  A closer look at Faster R-CNN for vehicle detection , 2016, 2016 IEEE Intelligent Vehicles Symposium (IV).

[11]  Sebastian Schmidtlein,et al.  Convolutional Neural Networks accurately predict cover fractions of plant species and communities in Unmanned Aerial Vehicle imagery , 2020, Remote Sensing in Ecology and Conservation.

[12]  R. Dubayah,et al.  Monitoring individual tree‐based change with airborne lidar , 2018, Ecology and evolution.

[13]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  E. Pinkard,et al.  Crown and stand characteristics of Eucalyptus nitens in response to initial spacing: implications for thinning , 2003 .

[15]  Terje Gobakken,et al.  Detection of small single trees in the forest–tundra ecotone using height values from airborne laser scanning , 2011 .

[16]  Lin Lei,et al.  Multi-scale object detection in remote sensing imagery with convolutional neural networks , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[17]  Jeremy Randle,et al.  Automated Mapping of Woody Debris over Harvested Forest Plantations Using UAVs, High-Resolution Imagery, and Machine Learning , 2019, Remote. Sens..

[18]  Michael A. Lefsky,et al.  Review of studies on tree species classification from remotely sensed data , 2016 .

[19]  Ye Zhang,et al.  A light and faster regional convolutional neural network for object detection in optical remote sensing images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[20]  Fan Zhang,et al.  TreeUNet: Adaptive Tree convolutional neural networks for subdecimeter aerial image segmentation , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[21]  Matthias Schubert,et al.  Automated Detection of Conifer Seedlings in Drone Imagery Using Convolutional Neural Networks , 2019, Remote. Sens..

[22]  S. Franklin Remote Sensing for Sustainable Forest Management , 2001 .

[23]  Mark O. Kimberley,et al.  The economic impact of optimising final stand density for structural saw log production on the value of the New Zealand plantation estate , 2017 .

[24]  Fabian Ewald Fassnacht,et al.  Convolutional Neural Networks enable efficient, accurate and fine-grained segmentation of plant species and communities from high-resolution UAV imagery , 2019, Scientific Reports.

[25]  Yannik Rist,et al.  Deep learning for multi-modal classification of cloud, shadow and land cover scenes in PlanetScope and Sentinel-2 imagery , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[26]  S LewMichael,et al.  Deep learning for visual understanding , 2016 .

[27]  Arko Lucieer,et al.  Sensor Correction of a 6-Band Multispectral Imaging Sensor for UAV Remote Sensing , 2012, Remote. Sens..

[28]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Jonathan P. Dash,et al.  Combining Airborne Laser Scanning and Aerial Imagery Enhances Echo Classification for Invasive Conifer Detection , 2017, Remote. Sens..

[30]  Philip E. Hulme,et al.  Assessing the utility of aerial imagery to quantify the density, age structure and spatial pattern of alien conifer invasions , 2019, Biological Invasions.

[31]  Giorgos Mallinis,et al.  On the Use of Unmanned Aerial Systems for Environmental Monitoring , 2018, Remote. Sens..

[32]  Arko Lucieer,et al.  Simplified radiometric calibration for UAS-mounted multispectral sensor , 2018 .

[33]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.

[34]  Chen Chen,et al.  One-Two-One Networks for Compression Artifacts Reduction in Remote Sensing , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[35]  Xueliang Zhang,et al.  Deep learning in remote sensing applications: A meta-analysis and review , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[36]  Juha Hyyppä,et al.  Outlook for the Next Generation’s Precision Forestry in Finland , 2014 .

[37]  Terje Gobakken,et al.  Automatic Detection of Small Single Trees in the Forest-Tundra Ecotone Using Airborne Laser Scanning , 2014, Remote. Sens..

[38]  Michael Förster,et al.  UAV data as alternative to field sampling to map woody invasive species based on combined Sentinel-1 and Sentinel-2 data , 2019, Remote Sensing of Environment.

[39]  Zhi-Hua Zhou,et al.  A brief introduction to weakly supervised learning , 2018 .

[40]  Tristan R. H. Goodbody,et al.  Assessing the status of forest regeneration using digital aerial photogrammetry and unmanned aerial systems , 2018 .

[41]  Shuyuan Yang,et al.  Pan-sharpening via deep metric learning , 2018 .

[42]  G. Castilla,et al.  Detection of Coniferous Seedlings in UAV Imagery , 2018, Forests.

[43]  Thomas Blaschke,et al.  Object based image analysis for remote sensing , 2010 .

[44]  Weijia Li,et al.  Deep Learning Based Oil Palm Tree Detection and Counting for High-Resolution Remote Sensing Images , 2016, Remote. Sens..

[45]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[46]  Nicolas Brown,et al.  Mapping dead forest cover using a deep convolutional neural network and digital aerial photography , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[47]  Fátima L. Benítez,et al.  A methodology for mapping native and invasive vegetation coverage in archipelagos , 2018 .

[48]  Erik Næsset,et al.  Using airborne laser scanning to monitor tree migration in the boreal–alpine transition zone , 2007 .

[49]  Terje Gobakken,et al.  Classifying tree and nontree echoes from airborne laser scanning in the forest–tundra ecotone , 2013 .

[50]  John R Moore,et al.  Influence of initial planting spacing and genotype on microfibril angle, wood density, fibre properties and modulus of elasticity in Pinus radiata D. Don corewood , 2009 .

[51]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[52]  Jonathan P. Dash,et al.  Early Detection of Invasive Exotic Trees Using UAV and Manned Aircraft Multispectral and LiDAR Data , 2019, Remote. Sens..

[53]  Shuang Wang,et al.  A deep learning framework for remote sensing image registration , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[54]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[55]  Bruce Talbot,et al.  Detection and Classification of Root and Butt-Rot (RBR) in Stumps of Norway Spruce Using RGB Images and Machine Learning , 2019, Sensors.

[56]  Peter I. Corke,et al.  Robotic detection and tracking of Crown-of-Thorns starfish , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[57]  J. L. Reid,et al.  How feasible are global forest restoration commitments? , 2020, Conservation Letters.

[58]  Congcong Li,et al.  Stacked Autoencoder-based deep learning for remote-sensing image classification: a case study of African land-cover mapping , 2016 .

[59]  Erik Næsset,et al.  Detection and Segmentation of Small Trees in the Forest-Tundra Ecotone Using Airborne Laser Scanning , 2016, Remote. Sens..

[60]  George E. Dahl,et al.  Artificial Intelligence-Based Breast Cancer Nodal Metastasis Detection: Insights Into the Black Box for Pathologists. , 2018, Archives of pathology & laboratory medicine.

[61]  Lloyd Windrim,et al.  Detection, Segmentation, and Model Fitting of Individual Tree Stems from Airborne Laser Scanning of Forests Using Deep Learning , 2020, Remote. Sens..