Deep Learning for Fusion of APEX Hyperspectral and Full-Waveform LiDAR Remote Sensing Data for Tree Species Mapping

Deep learning has been widely used to fuse multi-sensor data for classification. However, current deep learning architecture for multi-sensor data fusion might not always perform better than single data source, especially for the fusion of hyperspectral and light detection and ranging (LiDAR) remote sensing data for tree species mapping in complex, closed forest canopies. In this paper, we propose a new deep fusion framework to integrate the complementary information from hyperspectral and LiDAR data for tree species mapping. We also investigate the fusion of either “single-band” or multi-band (i.e., full-waveform) LiDAR with hyperspectral data for tree species mapping. Additionally, we provide a solution to estimate the crown size of tree species by the fusion of multi-sensor data. Experimental results on fusing real APEX hyperspectral and LiDAR data demonstrate the effectiveness of the proposed deep fusion framework. Compared to using only single data source or current deep fusion architecture, our proposed method yields improvements in overall and average classification accuracies ranging from 82.21% to 87.10% and 76.71% to 83.45%, respectively.

[1]  K. Itten,et al.  Fusion of imaging spectrometer and LIDAR data over combined radiative transfer models for forest canopy characterization , 2007 .

[2]  M. Cho,et al.  Classification of savanna tree species, in the Greater Kruger National Park region, by integrating hyperspectral and LiDAR data in a Random Forest data mining environment , 2012 .

[3]  B. Koetz,et al.  Discrimination of vegetation strata in a multi-layered Mediterranean forest ecosystem using height and intensity information derived from airborne laser scanning , 2010 .

[4]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[5]  Naoto Yokoya,et al.  Hyperspectral Pansharpening: A Review , 2015, IEEE Geoscience and Remote Sensing Magazine.

[6]  Giles M. Foody,et al.  Modelling geometric and misregistration error in airborne sensor data to enhance change detection , 2007 .

[7]  Kris Vandekerkhove,et al.  Bosreservaat Meerdaalwoud Everzwijnbad : monitoring van de dendrometrische gegevens en de vegetatie in steekproefcirkels en een kernvlakte , 2005 .

[8]  Aleksandra Pizurica,et al.  A comparison on multiple level features for fusion of hyperspectral and LiDAR data , 2017, 2017 Joint Urban Remote Sensing Event (JURSE).

[9]  Jon Atli Benediktsson,et al.  A Novel MKL Model of Integrating LiDAR Data and MSI for Urban Area Classification , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[10]  Liangpei Zhang,et al.  Fusion of Hyperspectral and LiDAR Data for Classification of Cloud-Shadow Mixed Remote Sensed Scene , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[11]  Naoto Yokoya,et al.  Advances in Hyperspectral Image and Signal Processing: A Comprehensive Overview of the State of the Art , 2017, IEEE Geoscience and Remote Sensing Magazine.

[12]  Antonio J. Plaza,et al.  Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[13]  Jon Atli Benediktsson,et al.  Extinction Profiles for the Classification of Remote Sensing Data , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[14]  Jon Atli Benediktsson,et al.  Classification of Remote Sensing Optical and LiDAR Data Using Extended Attribute Profiles , 2012, IEEE Journal of Selected Topics in Signal Processing.

[15]  Qian Du,et al.  Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion Contest , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[16]  Thomas Hofmann,et al.  Greedy Layer-Wise Training of Deep Networks , 2007 .

[17]  Peng Wang,et al.  An Adaptive Multi-Sensor Data Fusion Method Based on Deep Convolutional Neural Networks for Fault Diagnosis of Planetary Gearbox , 2017, Sensors.

[18]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[19]  Alexander Brenning,et al.  Using spectrotemporal indices to improve the fruit-tree crop classification accuracy , 2017 .

[20]  Jocelyn Chanussot,et al.  On the use of binary partition trees for the tree crown segmentation of tropical rainforest hyperspectral images , 2015 .

[21]  Uwe Weidner,et al.  Improvements of roof surface classification using hyperspectral and laser scanning data , 2005 .

[22]  Lorenzo Bruzzone,et al.  Extended profiles with morphological attribute filters for the analysis of hyperspectral data , 2010 .

[23]  S. Pauleit,et al.  Crown size and growing space requirement of common tree species in urban centres, parks, and forests , 2015 .

[24]  Naoto Yokoya,et al.  Hyperspectral Tree Species Classification of Japanese Complex Mixed Forest With the Aid of Lidar Data , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[25]  Xiao Xiang Zhu,et al.  Hyperspectral and LiDAR Data Fusion Using Extinction Profiles and Deep Convolutional Neural Network , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[26]  Jocelyn Chanussot,et al.  Support Vector Regression for the Estimation of Forest Stand Parameters Using Airborne Laser Scanning , 2011, IEEE Geoscience and Remote Sensing Letters.

[27]  Lorenzo Bruzzone,et al.  Fusion of Hyperspectral and LIDAR Remote Sensing Data for Classification of Complex Forest Areas , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[28]  Naoto Yokoya,et al.  Fusion of Hyperspectral and LiDAR Data for Landscape Visual Quality Assessment , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[29]  Gail P. Anderson,et al.  MODTRAN4 radiative transfer modeling for atmospheric correction , 1999, Optics & Photonics.

[30]  A. Hovi,et al.  LiDAR waveform features for tree species classification and their sensitivity to tree- and acquisition related parameters , 2016 .

[31]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[32]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[33]  Chengcui Zhang,et al.  A progressive morphological filter for removing nonground measurements from airborne LIDAR data , 2003, IEEE Trans. Geosci. Remote. Sens..

[34]  Seong-Whan Lee,et al.  Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis , 2014, NeuroImage.

[35]  Markus Hollaus,et al.  Total canopy transmittance estimated from small-footprint, full-waveform airborne LiDAR , 2017 .

[36]  Helmi Zulhaidi Mohd Shafri,et al.  A review on hyperspectral remote sensing for homogeneous and heterogeneous forest biodiversity assessment , 2010 .

[37]  Christian Debes,et al.  Automatic fusion and classification using random forests and features extracted with deep learning , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[38]  Wilfried Philips,et al.  Optimized feature fusion of LiDAR and hyperspectral data for tree species mapping in closed forest canopies , 2015, 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).

[39]  Loris Nanni,et al.  Handcrafted vs. non-handcrafted features for computer vision classification , 2017, Pattern Recognit..

[40]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[41]  Huanxin Zou,et al.  Synthetic Aperture Radar Target Recognition with Feature Fusion Based on a Stacked Autoencoder , 2017, Sensors.

[42]  Juha Hyyppä,et al.  An International Comparison of Individual Tree Detection and Extraction Using Airborne Laser Scanning , 2012, Remote. Sens..

[43]  Aleksandra Pizurica,et al.  Generalized Graph-Based Fusion of Hyperspectral and LiDAR Data Using Morphological Features , 2015, IEEE Geoscience and Remote Sensing Letters.

[44]  Rupesh Shrestha,et al.  Combining airborne hyperspectral and LiDAR data across local sites for upscaling shrubland structural information: lessons for HyspIRI. , 2015 .

[45]  Barbara Koch,et al.  Exploring full-waveform LiDAR parameters for tree species classification , 2011, Int. J. Appl. Earth Obs. Geoinformation.

[46]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[47]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[48]  Michael A. Lefsky,et al.  Review of studies on tree species classification from remotely sensed data , 2016 .

[49]  Gang Wang,et al.  Deep Learning-Based Classification of Hyperspectral Data , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[50]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[51]  A. Viera,et al.  Understanding interobserver agreement: the kappa statistic. , 2005, Family medicine.

[52]  Joachim Hill,et al.  Fusion of full-waveform lidar and imaging spectroscopy remote sensing data for the characterization of forest stands , 2013 .

[53]  Jie Geng,et al.  High-Resolution SAR Image Classification via Deep Convolutional Autoencoders , 2015, IEEE Geoscience and Remote Sensing Letters.

[54]  Jiquan Ngiam,et al.  Sparse Filtering , 2011, NIPS.

[55]  Åsa Persson,et al.  Species identification of individual trees by combining high resolution LiDAR data with multi‐spectral images , 2008 .

[56]  Pedram Ghamisi,et al.  Hyperspectral and LiDAR Fusion Using Extinction Profiles and Total Variation Component Analysis , 2017, IEEE Transactions on Geoscience and Remote Sensing.