Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks

In this work, we describe a new, general, and efficient method for unstructured point cloud labeling. As the question of efficiently using deep Convolutional Neural Networks (CNNs) on 3D data is still a pending issue, we propose a framework which applies CNNs on multiple 2D image views (or snapshots) of the point cloud. The approach consists in three core ideas. (i) We pick many suitable snapshots of the point cloud. We generate two types of images: a Red-Green-Blue (RGB) view and a depth composite view containing geometric features. (ii) We then perform a pixel-wise labeling of each pair of 2D snapshots using fully convolutional networks. Different architectures are tested to achieve a profitable fusion of our heterogeneous inputs. (iii) Finally, we perform fast back-projection of the label predictions in the 3D space using efficient buffering to label every 3D point. Experiments show that our method is suitable for various types of point clouds such as Lidar or photogrammetric data.

[1]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Reinhard Klein,et al.  Efficient RANSAC for Point‐Cloud Shape Detection , 2007, Comput. Graph. Forum.

[3]  Andrew E. Johnson,et al.  Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Florent Lafarge,et al.  Creating Large-Scale City Models from 3D-Point Clouds: A Robust Approach with Hybrid Representation , 2012, International Journal of Computer Vision.

[5]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[6]  Alexandre Boulch,et al.  Piecewise‐Planar 3D Reconstruction with Edge and Corner Regularization , 2014, Comput. Graph. Forum.

[7]  C. Brenner,et al.  URBAN GIS FROM LASER ALTIMETER AND 2 D MAP DATA , 1998 .

[8]  Nico Blodow,et al.  Fast geometric point labeling using conditional random fields , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[9]  Dieter Fox,et al.  Unsupervised feature learning for 3D scene labeling , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[10]  C. Briese,et al.  A NEW METHOD FOR BUILDING EXTRACTION IN URBAN AREAS FROM HIGH-RESOLUTION LIDAR DATA , 2002 .

[11]  Benjamin Graham,et al.  Spatially-sparse convolutional neural networks , 2014, ArXiv.

[12]  Marc Pollefeys,et al.  Semantic3D.net: A new Large-scale Point Cloud Classification Benchmark , 2017, ArXiv.

[13]  C. Mallet,et al.  AIRBORNE LIDAR FEATURE SELECTION FOR URBAN CLASSIFICATION USING RANDOM FORESTS , 2009 .

[14]  Roberto Manduchi,et al.  Supervised Parametric Classification of Aerial LiDAR Data , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[15]  Vladimir G. Kim,et al.  Shape-based recognition of 3D point clouds in urban environments , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[16]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Markus Gerke,et al.  The ISPRS benchmark on urban object classification and 3D building reconstruction , 2012 .

[18]  Jan Dirk Wegner,et al.  Mind the Gap: Modeling Local and Global Context in (Road) Networks , 2014, GCPR.

[19]  Zoltan-Csaba Marton,et al.  On Fast Surface Reconstruction Methods for Large and Noisy Datasets , 2009, IEEE International Conference on Robotics and Automation.

[20]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Rafael Grompone von Gioi,et al.  Fast plane detection in disparity maps , 2010, 2010 IEEE International Conference on Image Processing.

[22]  Florent Lafarge,et al.  Hybrid multi-view reconstruction by Jump-Diffusion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Takeo Kanade,et al.  A Multiple-Baseline Stereo , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Alexandre Boulch,et al.  Deep Learning for Robust Normal Estimation in Unstructured Point Clouds , 2016, Comput. Graph. Forum.

[26]  Konrad Schindler,et al.  FAST SEMANTIC SEGMENTATION OF 3D POINT CLOUDS WITH STRONGLY VARYING DENSITY , 2016 .

[27]  Bertrand Le Saux,et al.  Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks , 2016, ACCV.

[28]  C. Brenner,et al.  3D URBAN GIS FROM LASER ALTIMETER AND 2D MAP DATA , 1997 .

[29]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[30]  H. Maas.  THE POTENTIAL OF HEIGHT TEXTURE MEASURES FOR THE SEGMENTATION OF AIRBORNE LASERSCANNER DATA , 1999 .

[31]  Federico Tombari,et al.  Unique Signatures of Histograms for Local Surface Description , 2010, ECCV.

[32]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Leif Kobbelt,et al.  Identifying Style of 3D Shapes using Deep Metric Learning , 2016, Comput. Graph. Forum.

[34]  G. Sithole,et al.  Recognising structure in laser scanning point clouds , 2004 .