Multi-task fully convolutional network for tree species mapping in dense forests using small training hyperspectral data

This work proposes a multi-task fully convolutional architecture for tree species mapping in dense forests from sparse and scarce polygon-level annotations using hyperspectral UAV-borne data. Our model implements a partial loss function that enables dense tree semantic labeling outcomes from non-dense training samples, and a distance regression complementary task that enforces tree crown boundary constraints and substantially improves the model performance. Our multi-task architecture uses a shared backbone network that learns common representations for both tasks and two task-specific decoders, one for the semantic segmentation output and one for the distance map regression. We report that introducing the ∗Corresponding author Email addresses: lauracue@aluno.puc-rio.br (Laura Elena Cué La Rosa), sothec@mcmaster.ca (Camile Sothe), raul@ele.puc-rio.br (Raul Queiroz Feitosa), almeida@dsr.inpe.br (Cláudia Maria de Almeida), marcos.schimalski@udesc.br (Marcos Benedito Schimalski), dariobo@br.ibm.com (Dario Augusto Borges Oliveira) 1 ar X iv :2 10 6. 00 79 9v 2 [ cs .C V ] 6 S ep 2 02 1 complementary task boosts the semantic segmentation performance compared to the single-task counterpart in up to 11% reaching an average user’s accuracy of 88.63% and an average producer’s accuracy of 88.59%, achieving state-of-art performance for tree species classification in tropical forests.

[1]  Aniruddha Ghosh,et al.  A framework for mapping tree species combining hyperspectral and LiDAR data: Role of selected classifiers and sensor across three spatial scales , 2014, Int. J. Appl. Earth Obs. Geoinformation.

[2]  Eija Honkavaara,et al.  A Novel Deep Learning Method to Identify Single Tree Species in UAV-Based Hyperspectral Images , 2020, Remote. Sens..

[3]  Yuri Boykov,et al.  Normalized Cut Loss for Weakly-Supervised CNN Segmentation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Y. Shimabukuro,et al.  Mapping tree species in tropical seasonal semi-deciduous forests with hyperspectral and multispectral data , 2016 .

[5]  Michele Volpi,et al.  Dense Semantic Labeling of Subdecimeter Resolution Images With Convolutional Neural Networks , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[6]  Xiuping Jia,et al.  Hyperspectral Image Classification Using Convolutional Neural Networks and Multiple Feature Learning , 2018, Remote. Sens..

[7]  Xin Shen,et al.  Tree-Species Classification in Subtropical Forests Using Airborne Hyperspectral and LiDAR Data , 2017, Remote. Sens..

[8]  Erle C. Ellis,et al.  ForestGEO: Understanding forest diversity and dynamics through a global observatory network , 2021, Biological Conservation.

[9]  S. Reed,et al.  Urgent need for warming experiments in tropical forests , 2015, Global change biology.

[10]  Fabian Ewald Fassnacht,et al.  Convolutional Neural Networks enable efficient, accurate and fine-grained segmentation of plant species and communities from high-resolution UAV imagery , 2019, Scientific Reports.

[11]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Lindi J. Quackenbush,et al.  A comparison of three methods for automatic tree crown detection and delineation from high spatial resolution imagery , 2011 .

[13]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Russell G. Congalton,et al.  Assessing the accuracy of remotely sensed data : principles and practices , 1998 .

[15]  Gregory Asner,et al.  Tree Species Abundance Predictions in a Tropical Agricultural Landscape with a Supervised Classification Model and Imbalanced Data , 2016, Remote. Sens..

[16]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[17]  Jocelyn Chanussot,et al.  On the use of binary partition trees for the tree crown segmentation of tropical rainforest hyperspectral images , 2015 .

[18]  Xiao Xiang Zhu,et al.  Semantic Segmentation of Remote Sensing Images With Sparse Annotations , 2021, IEEE Geoscience and Remote Sensing Letters.

[19]  Peter Caccetta,et al.  ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[20]  Michele Dalponte,et al.  Delineation of Individual Tree Crowns from ALS and Hyperspectral data: a comparison among four methods , 2015 .

[21]  Sergio Benini,et al.  Deep Learning Meets Hyperspectral Image Analysis: A Multidisciplinary Review , 2019, J. Imaging.

[22]  Y. Shimabukuro,et al.  Tree species classification in tropical forests using visible to shortwave infrared WorldView-3 images and texture analysis , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[23]  George C. Hurtt,et al.  An efficient, multi-layered crown delineation algorithm for mapping individual tree structure across multiple ecosystems , 2014 .

[24]  Ilkka Pölönen,et al.  Tree Species Identification Using 3D Spectral Data and 3D Convolutional Neural Network , 2018, 2018 9th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[25]  V. Liesenberg,et al.  Comparative performance of convolutional neural network, weighted and conventional support vector machine and random forest for classifying tree species using hyperspectral and photogrammetric data , 2020, GIScience & Remote Sensing.

[26]  Evandro Orfanó Figueiredo,et al.  Individual tree detection and species classification of Amazonian palms using UAV images and deep learning , 2020, Forest Ecology and Management.

[27]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Roberta Kwok,et al.  Ecology’s remote-sensing revolution , 2018, Nature.

[29]  David E. Knapp,et al.  Operational Tree Species Mapping in a Diverse Tropical Forest with Airborne Imaging Spectroscopy , 2015, PloS one.

[30]  Michele Dalponte,et al.  Tree Species Classification in a Highly Diverse Subtropical Forest Integrating UAV-Based Photogrammetric Point Cloud and Hyperspectral Data , 2019, Remote. Sens..

[31]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[32]  Xueliang Zhang,et al.  Deep learning in remote sensing applications: A meta-analysis and review , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[33]  Michael Crawshaw,et al.  Multi-Task Learning with Deep Neural Networks: A Survey , 2020, ArXiv.

[34]  Bo Du,et al.  Deep Learning for Remote Sensing Data: A Technical Tutorial on the State of the Art , 2016, IEEE Geoscience and Remote Sensing Magazine.

[35]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Sebastian Schmidtlein,et al.  Mapping forest tree species in high resolution UAV-based RGB-imagery by means of convolutional neural networks , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[37]  Gregory Asner,et al.  Tree Species Discrimination in Tropical Forests Using Airborne Imaging Spectroscopy , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Heikki Saari,et al.  Processing and Assessment of Spectrometric, Stereoscopic Imagery Collected Using a Lightweight UAV Spectral Camera for Precision Agriculture , 2013, Remote. Sens..

[39]  Martin Brandt,et al.  An unexpectedly large count of trees in the West African Sahara and Sahel , 2020, Nature.

[40]  Pedro Higuchi,et al.  Influência de variáveis ambientais sobre o padrão estrutural e florístico do componente arbóreo, em um fragmento de Floresta Ombrófila Mista Montana em Lages, SC , 2012 .

[41]  Ying Li,et al.  Spectral-Spatial Classification of Hyperspectral Imagery with 3D Convolutional Neural Network , 2017, Remote. Sens..

[42]  Carola-Bibiane Schönlieb,et al.  A graph cut approach to 3D tree delineation, using integrated airborne LiDAR and hyperspectral imagery , 2017, ArXiv.

[43]  Sassan Saatchi,et al.  Lidar detection of individual tree size in tropical forests , 2016 .

[44]  E. Honkavaara,et al.  Geometric Calibration of a Hyperspectral Frame Camera , 2016 .

[45]  O. Phillips,et al.  Using the U‐net convolutional network to map forest types and disturbance in the Atlantic rainforest with very high resolution images , 2019, Remote Sensing in Ecology and Conservation.

[46]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Qiang Yang,et al.  An Overview of Multi-task Learning , 2018 .

[48]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[49]  Dar A. Roberts,et al.  Species-Level Differences in Hyperspectral Metrics among Tropical Rainforest Trees as Determined by a Tree-Based Classifier , 2012, Remote. Sens..

[50]  Ana Cristina Murillo,et al.  Coral-Segmentation: Training Dense Labeling Models with Sparse Ground Truth , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[51]  C. Y. Kwok,et al.  Characterizing and classifying urban tree species using bi-monthly terrestrial hyperspectral images in Hong Kong , 2021, ISPRS Journal of Photogrammetry and Remote Sensing.

[52]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[53]  Andreas Dengel,et al.  Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networks , 2017, 2019 IEEE International Conference on Image Processing (ICIP).

[54]  Janet Franklin,et al.  A Convolutional Neural Network Classifier Identifies Tree Species in Mixed-Conifer Forest from Hyperspectral Imagery , 2019, Remote. Sens..

[55]  P. Ferreira,et al.  DISSIMILARIDADE FLORÍSTICA E ESPÉCIES INDICADORAS DE FLORESTA OMBRÓFILA MISTA E ECÓTONOS NO PLANALTO SUL CATARINENSE , 2015 .

[56]  Bin Zhang,et al.  Three-dimensional convolutional neural network model for tree species classification using airborne hyperspectral images , 2020, Remote Sensing of Environment.

[57]  Erxue Chen,et al.  Improved Prototypical Network Model for Forest Species Classification in Complex Stand , 2020, Remote. Sens..

[58]  Huchuan Lu,et al.  Deep Mutual Learning , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Bernt Schiele,et al.  Simple Does It: Weakly Supervised Instance and Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  M. Onishi,et al.  Explainable identification and mapping of trees using UAV RGB image and deep learning , 2021, Scientific Reports.

[62]  Samia Boukir,et al.  Exploring issues of training data imbalance and mislabelling on random forest performance for large area land cover classification using the ensemble margin , 2015 .

[63]  Raul Queiroz Feitosa,et al.  Applying Fully Convolutional Architectures for Semantic Segmentation of a Single Tree Species in Urban Environment on High Resolution UAV Optical Imagery , 2020, Sensors.

[64]  Maitiniyazi Maimaitijiang,et al.  Urban Tree Species Classification Using a WorldView-2/3 and LiDAR Data Fusion Approach and Deep Learning , 2019, Sensors.

[65]  Costas Armenakis,et al.  Individual tree species identification using Dense Convolutional Network (DenseNet) on multitemporal RGB images from UAV , 2020, Journal of Unmanned Vehicle Systems.

[66]  Liang Liu,et al.  Scribble-Supervised Segmentation of Aerial Building Footprints Using Adversarial Learning , 2018, IEEE Access.

[67]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[68]  Christopher J. Post,et al.  Automated tree-crown and height detection in a young forest plantation using mask region-based convolutional neural network (Mask R-CNN) , 2021, ISPRS Journal of Photogrammetry and Remote Sensing.

[69]  Guidelines for Using the IUCN Red List Categories and Criteria , 2005 .

[70]  D. Roberts,et al.  Hyperspectral discrimination of tropical rain forest tree species at leaf to crown scales , 2005 .

[71]  O. Phillips,et al.  Individual tree crown delineation in a highly diverse tropical forest using very high resolution satellite images , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[72]  Sakari Tuominen,et al.  Tree species classification from airborne hyperspectral and LiDAR data using 3D convolutional neural networks , 2021, Remote Sensing of Environment.

[73]  Gabriele Moser,et al.  Improving Maps from CNNs Trained with Sparse, Scribbled Ground Truths Using Fully Connected CRFs , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[74]  Michael A. Lefsky,et al.  Review of studies on tree species classification from remotely sensed data , 2016 .

[75]  Sheng Tang,et al.  Boundary Perception Guidance: A Scribble-Supervised Semantic Segmentation Approach , 2019, IJCAI.

[76]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[77]  Rodolfo G. Lotte,et al.  Accurate mapping of Brazil nut trees (Bertholletia excelsa) in Amazonian forests using WorldView-3 satellite images and convolutional neural networks , 2021, Ecol. Informatics.