TSCMDL: Multimodal Deep Learning Framework for Classifying Tree Species Using Fusion of 2-D and 3-D Features

Accurate tree species information is a prerequisite for forest resource management. Combining light detection and ranging (LiDAR) and image data is one main method of tree species classification. Traditional machine learning methods rely on expert knowledge to calculate a large number of feature parameters. Deep learning technology can directly use the original image and point cloud data to classify tree species. However, data with different patterns require the use of different types of deep learning methods. In this study, a tree species classification multimodal deep learning (TSCMDL) that fuses 2-D and 3-D features was constructed and then used to combine data from multiple sources for tree species classification. This framework uses an improved version of the PointMLP model as its backbone network and uses ResNet50 and PointMLP networks to extract the image features and point cloud features, respectively. The proposed framework was tested using unmanned aerial vehicle LiDAR (UAV LiDAR) data and red, green, blue (RGB) orthophotos. The results showed that the accuracy of the tree species classification using the TSCMDL framework was 98.52%, which was 4.02% higher than that based on point cloud features only. In addition, when the same hyperparameters were used for training the model, the efficiency of the model training was not significantly lower than for models based on point cloud features only. The proposed multimodal deep learning framework extracts features directly from the original data and integrates them effectively, thus avoiding manual feature screening and achieving more accurate classification. The feature extraction network used in the TSCMDL framework can be replaced by other suitable frameworks and has strong application potential.

[1]  Juan Carlos Niebles,et al.  ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Zeng-yuan Li,et al.  Tree Species Classification Using Ground-Based LiDAR Data by Various Point Cloud Deep Learning Methods , 2022, Remote. Sens..

[3]  Huaguo Huang,et al.  Individual Tree Species Classification Using the Pointwise MLP-Based Point Cloud Deep Learning Method , 2022, IECF 2022.

[4]  Weiqi Zhou,et al.  Individual tree segmentation and tree species classification in subtropical broadleaf forests using UAV-based LiDAR, hyperspectral, and ultrahigh-resolution RGB data , 2022, Remote Sensing of Environment.

[5]  Haozhe Zhong,et al.  Identification of tree species based on the fusion of UAV hyperspectral image and LiDAR data in a coniferous and broad-leaved mixed forest in Northeast China , 2022, Frontiers in Plant Science.

[6]  Huaguo Huang,et al.  Tree Species Classification of Backpack Laser Scanning Data Using the PointNet++ Point Cloud Deep Learning Method , 2022, Remote. Sens..

[7]  Guoqi Chai,et al.  ACE R-CNN: An Attention Complementary and Edge Detection-Based Instance Segmentation Algorithm for Individual Tree Species Identification Using UAV RGB Images and LiDAR Data , 2022, Remote. Sens..

[8]  Xiaodan Wu,et al.  Characterizing the Effect of Spatial Heterogeneity and the Deployment of Sampled Plots on the Uncertainty of Ground “Truth” on a Coarse Grid Scale: Case Study for Near‐Infrared (NIR) Surface Reflectance , 2022, Journal of Geophysical Research: Atmospheres.

[9]  M. Wong,et al.  Tropical Species Classification with Structural Traits Using Handheld Laser Scanning Data , 2022, Remote. Sens..

[10]  Amin Muhammad Shoib,et al.  A Review on Methods and Applications in Multimodal Deep Learning , 2022, ACM Trans. Multim. Comput. Commun. Appl..

[11]  Y. Fu,et al.  Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework , 2022, ICLR.

[12]  Zhengrong Li,et al.  A Convex Hull-Based Feature Descriptor for Learning Tree Species Classification From ALS Point Clouds , 2021, IEEE Geoscience and Remote Sensing Letters.

[13]  Lihu Dong,et al.  A Hierarchical Region-Merging Algorithm for 3-D Segmentation of Individual Trees Using UAV-LiDAR Point Clouds , 2022, IEEE Transactions on Geoscience and Remote Sensing.

[14]  L. Bruzzone,et al.  An Approach Based on Deep Learning for Tree Species Classification in LiDAR Data Acquired in Mixed Forest , 2022, IEEE Geoscience and Remote Sensing Letters.

[15]  Baoxin Hu,et al.  CNN-Based Individual Tree Species Classification Using High-Resolution Satellite Imagery and Airborne LiDAR Data , 2021, Forests.

[16]  Zhengjun Liu,et al.  Classification of Typical Tree Species in Laser Point Cloud Based on Deep Learning , 2021, Remote. Sens..

[17]  Elizaveta K. Sakharova,et al.  Issues of Tree Species Classification from LiDAR Data Using Deep Learning Model , 2021, Advances in Neural Computation, Machine Learning, and Cognitive Research V.

[18]  Maxwell B. Joseph,et al.  Fusion neural networks for plant classification: learning to combine RGB, hyperspectral, and lidar data , 2021, PeerJ.

[19]  Andrew J. Sánchez Meador,et al.  Adjudicating Perspectives on Forest Structure: How Do Airborne, Terrestrial, and Mobile Lidar-Derived Estimates Compare? , 2021, Remote. Sens..

[20]  Sakari Tuominen,et al.  Tree species classification from airborne hyperspectral and LiDAR data using 3D convolutional neural networks , 2021, Remote Sensing of Environment.

[21]  Zhengjun Liu,et al.  Tree species classification of LiDAR data based on 3D deep learning , 2021 .

[22]  T. Kneib,et al.  Predicting Tree Species From 3D Laser Scanning Point Clouds Using Deep Learning , 2021, Frontiers in Plant Science.

[23]  Mohammed Bennamoun,et al.  Deep Learning for 3D Point Clouds: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  George Vosselman,et al.  Silvi-Net - A dual-CNN approach for combined classification of tree species and standing dead trees from remote sensing data , 2021, Int. J. Appl. Earth Obs. Geoinformation.

[25]  Y. Malhi,et al.  Tree species classification using structural features derived from terrestrial laser scanning , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[26]  C. Hopkinson,et al.  See the forest and the trees: Effective machine and deep learning algorithms for wood filtering and tree species classification from terrestrial laser scanning , 2020 .

[27]  Nicholas C. Coops,et al.  Tree species classification using UAS-based digital aerial photogrammetry point clouds and multispectral imageries in subtropical natural forests , 2020, Int. J. Appl. Earth Obs. Geoinformation.

[28]  Bin Zhang,et al.  Three-dimensional convolutional neural network model for tree species classification using airborne hyperspectral images , 2020, Remote Sensing of Environment.

[29]  Costas Armenakis,et al.  Individual tree species identification using Dense Convolutional Network (DenseNet) on multitemporal RGB images from UAV , 2020, Journal of Unmanned Vehicle Systems.

[30]  Xiaoli Zhang,et al.  Classification of planted forest species in southern China with airborne hyperspectral and LiDAR data , 2020 .

[31]  Yong Pang,et al.  Individual Tree Classification Using Airborne LiDAR and Hyperspectral Data in a Natural Mixed Forest of Northeast China , 2020, Forests.

[32]  Juha Hyyppä,et al.  Accurate derivation of stem curve and volume using backpack mobile laser scanning , 2020, ISPRS Journal of Photogrammetry and Remote Sensing.

[33]  Marco Heurich,et al.  Large-Scale Mapping of Tree Species and Dead Trees in Šumava National Park and Bavarian Forest National Park Using Lidar and Multispectral Imagery , 2020, Remote. Sens..

[34]  Marco Heurich,et al.  Improving LiDAR-based tree species mapping in Central European mixed forests using multi-temporal digital aerial colour-infrared photographs , 2020, Int. J. Appl. Earth Obs. Geoinformation.

[35]  Raul Queiroz Feitosa,et al.  Applying Fully Convolutional Architectures for Semantic Segmentation of a Single Tree Species in Urban Environment on High Resolution UAV Optical Imagery , 2020, Sensors.

[36]  Xiaoli Zhang,et al.  Object-Based Tree Species Classification Using Airborne Hyperspectral Images and LiDAR Data , 2019, Forests.

[37]  Hongsheng Zhang,et al.  Characterizing Tree Species of a Tropical Wetland in Southern China at the Individual Tree Level Based on Convolutional Neural Network , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[38]  Janet Franklin,et al.  A Convolutional Neural Network Classifier Identifies Tree Species in Mixed-Conifer Forest from Hyperspectral Imagery , 2019, Remote. Sens..

[39]  Jonathan Li,et al.  Rapid Urban Roadside Tree Inventory Using a Mobile Laser Scanning System , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[40]  Xiaoyong Shen,et al.  STD: Sparse-to-Dense 3D Object Detector for Point Cloud , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Costas Armenakis,et al.  RESNET-BASED TREE SPECIES CLASSIFICATION USING UAV IMAGES , 2019, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[42]  Michele Dalponte,et al.  Tree Species Classification in a Highly Diverse Subtropical Forest Integrating UAV-Based Photogrammetric Point Cloud and Hyperspectral Data , 2019, Remote. Sens..

[43]  Sergio Benini,et al.  Deep Learning Meets Hyperspectral Image Analysis: A Multidisciplinary Review , 2019, J. Imaging.

[44]  Wenge Ni-Meister,et al.  Machine Learning Techniques for Tree Species Classification Using Co-Registered LiDAR and Hyperspectral Data , 2019, Remote. Sens..

[45]  Guang Yang,et al.  Tree Species Classification by Employing Multiple Features Acquired from Integrated Sensors , 2019, J. Sensors.

[46]  Hamid Hamraz,et al.  Deep learning for conifer/deciduous classification of airborne LiDAR 3D point clouds representing individual trees , 2018, ISPRS Journal of Photogrammetry and Remote Sensing.

[47]  Xuehua Liu,et al.  A Review: Individual Tree Species Classification Using Integrated Airborne LiDAR and Optical Imagery with a Focus on the Urban Environment , 2018, Forests.

[48]  Arnt-Børre Salberg,et al.  Tree species classification in Norway from airborne hyperspectral and airborne laser scanning data , 2018 .

[49]  Ming Cheng,et al.  Tree Classification in Complex Forest Point Clouds Based on Deep Learning , 2017, IEEE Geoscience and Remote Sensing Letters.

[50]  Lei Gao,et al.  Signal Processing: Image Communication , 2022 .

[51]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[52]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Wuming Zhang,et al.  An Easy-to-Use Airborne LiDAR Data Filtering Method Based on Cloth Simulation , 2016, Remote. Sens..

[54]  Yi Lin,et al.  Tree species classification based on explicit tree structure feature parameters derived from static terrestrial laser scanning data , 2016 .

[55]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Qi Zhang,et al.  Deep learning-based tree classification using mobile LiDAR data , 2015 .