Classification of ALS Point Clouds Using End-to-End Deep Learning

Deep learning, referring to artificial neural networks with multiple layers, is widely used for classification tasks in many disciplines including computer vision. The most popular type is the Convolutional Neural Network (CNN), commonly applied to 2D image data. However, CNNs are difficult to adapt to irregular data like point clouds. PointNet, on the other hand, has enabled the derivation of features based on the geometric distribution of a set of points in nD-space utilising a neural network. We use PointNet on multiple scales to automatically learn a representation of local neighbourhoods in an end-to-end fashion, which is optimised for semantic labelling on 3D point clouds acquired by Airborne Laser Scanning (ALS). The results are comparable to those using manually crafted features, suggesting a successful representation of these neighbourhoods. On the ISPRS 3D Semantic Labelling benchmark, we achieve 80.6% overall accuracy, a mid-field result. Investigation on a bigger dataset, namely the 2011 ALS point cloud of the federal state of Vorarlberg, shows overall accuracies of up to 95.8% over large-scale built-up areas. Lower accuracy is achieved for the separation of low vegetation and ground points, presumably because of invalid assumptions about the distribution of classes in space, especially in high alpine regions. We conclude that the method of the end-to-end system, allowing training on a big variety of classification problems without the need for expert knowledge about neighbourhood features can also successfully be applied to single-point-based classification of ALS point clouds.

[1]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[2]  S. Schmohl,et al.  SUBMANIFOLD SPARSE CONVOLUTIONAL NETWORKS FOR SEMANTIC SEGMENTATION OF LARGE-SCALE ALS POINT CLOUDS , 2019 .

[3]  Michael Felsberg,et al.  Deep Projective 3D Semantic Segmentation , 2017, CAIP.

[4]  Silvio Savarese,et al.  SEGCloud: Semantic Segmentation of 3D Point Clouds , 2017, 2017 International Conference on 3D Vision (3DV).

[5]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Fabio Remondino,et al.  A REVIEW OFPOINT CLOUDS SEGMENTATION AND CLASSIFICATION ALGORITHMS , 2017 .

[7]  Jing Huang,et al.  Point cloud labeling using 3D Convolutional Neural Network , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[8]  Norbert Pfeifer,et al.  Georeferenced Point Clouds: A Survey of Features and Point Cloud Management , 2013, ISPRS Int. J. Geo Inf..

[9]  Monika Sester,et al.  EXPLORING ALS AND DIM DATA FOR SEMANTIC SEGMENTATION USING CNNS , 2018, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences.

[10]  Konrad Schindler,et al.  FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY , 2016 .

[11]  Alfred Stein,et al.  Deep Fully Convolutional Networks for the Detection of Informal Settlements in VHR Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[12]  Uwe Soergel,et al.  HIERARCHICAL HIGHER ORDER CRF FOR THE CLASSIFICATION OF AIRBORNE LIDAR POINT CLOUDS IN URBAN AREAS , 2016 .

[13]  Clément Mallet.  Analyse des données lidar aéroportées à Retour d'Onde Complète pour la cartographie des milieux urbains. (Analysis of Full-Waveform lidar data for urban area mapping) , 2010 .

[14]  Wei Huang,et al.  A Convolutional Neural Network-Based 3D Semantic Labeling Method for ALS Point Clouds , 2017, Remote. Sens..

[15]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[16]  Alexandre Boulch,et al.  Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks , 2017, 3DOR@Eurographics.

[17]  Ruibin Zhao,et al.  Classifying airborne LiDAR point clouds via deep features learned by a multi-scale convolutional neural network , 2018, Int. J. Geogr. Inf. Sci..

[18]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[19]  Thomas A. Funkhouser,et al.  Semantic Scene Completion from a Single Depth Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Matthias Nießner,et al.  ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Gregory Asner,et al.  Spatially-Explicit Testing of a General Aboveground Carbon Density Estimation Model in a Western Amazonian Forest Using Airborne LiDAR , 2015, Remote. Sens..

[23]  J. D. Wegner,et al.  Large-scale supervised learning for 3D point cloud labeling: Semantic3d.net , 2018 .

[24]  J. Niemeyer,et al.  Contextual classification of lidar data and building object detection in urban areas , 2014 .

[25]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[26]  Bernhard Höfle,et al.  HELIOS: A MULTI-PURPOSE LIDAR SIMULATION FRAMEWORK FOR RESEARCH, PLANNING AND TRAINING OF LASER SCANNING OPERATIONS WITH AIRBORNE, GROUND-BASED MOBILE AND STATIONARY PLATFORMS , 2016 .

[27]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[28]  Carl Salvaggio,et al.  A Fully Convolutional Network for Semantic Labeling of 3D Point Clouds , 2017, ISPRS Journal of Photogrammetry and Remote Sensing.

[29]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[30]  Norbert Pfeifer,et al.  Classification of image matching point clouds over an urban area , 2018 .

[31]  Martin Simonovsky,et al.  Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[32]  Johannes Heinzel,et al.  Detecting Tree Stems from Volumetric TLS Data in Forest Environments with Rich Understory , 2016, Remote. Sens..

[33]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[34]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[35]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  S. J. Oude Elberink,et al.  FULLY CONVOLUTIONAL NETWORKS FOR GROUND CLASSIFICATION FROM LIDAR POINT CLOUDS , 2018 .

[37]  Martin Weinmann,et al.  USING MULTI-SCALE FEATURES FOR THE 3D SEMANTIC LABELING OFAIRBORNE LASER SCANNING DATA , 2017 .

[38]  C. Mallet,et al.  AIRBORNE LIDAR FEATURE SELECTION FOR URBAN CLASSIFICATION USING RANDOM FORESTS , 2009 .

[39]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[40]  Boris Jutzi,et al.  Feature relevance assessment for the semantic interpretation of 3D point cloud data , 2013 .

[41]  Wolfgang Wagner,et al.  WAVEFORM ANALYSIS TECHNIQUES IN AIRBORNE LASER SCANNING , 2007 .

[42]  Michael Cramer,et al.  The DGPF-Test on Digital Airborne Camera Evaluation - Over- view and Test Design , 2010 .

[43]  G. Vosselman.  Point cloud segmentation for urban scene classification , 2013 .

[44]  Xiangyun Hu,et al.  Deep-Learning-Based Classification for DTM Extraction from ALS Point Cloud , 2016, Remote. Sens..

[45]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[46]  Stefan Hinz,et al.  Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers , 2015 .

[47]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Laurens van der Maaten,et al.  3D Semantic Segmentation with Submanifold Sparse Convolutional Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.