MHNet: Multiscale Hierarchical Network for 3D Point Cloud Semantic Segmentation

Point cloud semantic segmentation is a challenging task in 3D understanding due to its disorder, unstructured and nonuniform density. Currently, most methods focus on network design and feature extraction. However, it is difficult to capture the point cloud features of complex objects comprehensively and accurately. In this paper, we propose a multiscale hierarchical network (MHNet) for 3D point cloud semantic segmentation. First, a hierarchical point cloud feature extraction structure is constructed to learn multiscale local region features. Then, these local features are subjected to feature propagation to obtain the features of the entire point set for pointwise label prediction. To take fully advantage of the correlations of propagated information between the different scale coarse layers and the original points, the local features of each scale are characterized by feature propagation to obtain the features of the original point clouds at the corresponding scale. The global features propagated from different scales are integrated to constitute the final features of the input point clouds. The concatenated multiscale hierarchical features, including both local features and global features, can better predict the segmentation probability of each point cloud. Finally, the predicted segmentation results are optimized using the conditional random filed (CRF) with a spatial consistency constraint. The efficiency of MHNet is evaluated on two 3D datasets (S3DIS and ScanNet), and the results show performance comparable or superior to the state-of-the-art on both datasets.

[1]  José García Rodríguez,et al.  A Review on Deep Learning Techniques Applied to Semantic Segmentation , 2017, ArXiv.

[2]  Ulrich Neumann,et al.  SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Shu Liu,et al.  Associatively Segmenting Instances and Semantics in Point Clouds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Lei Wang,et al.  MSNet: Multi-Scale Convolutional Network for Point Cloud Classification , 2018, Remote. Sens..

[5]  Ulrich Neumann,et al.  3D point cloud object detection with multi-view convolutional neural network , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[6]  Jing Huang,et al.  Vehicle detection in urban point clouds with orthogonal-view convolutional neural network , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[7]  Silvio Savarese,et al.  3D Semantic Parsing of Large-Scale Indoor Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[9]  Bastian Leibe,et al.  Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[10]  Ulrich Neumann,et al.  IPDC: Iterative part-based dense correspondence between point clouds , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[11]  Jing Huang,et al.  Detecting Objects in Scene Point Cloud: A Combinational Approach , 2013, 2013 International Conference on 3D Vision.

[12]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Javier Ruiz Hidalgo,et al.  Segmentation-based Multi-scale Edge Extraction to Measure the Persistence of Features in Unorganized Point Clouds , 2017, VISIGRAPP.

[14]  Wei Wu,et al.  PointCNN: Convolution On X-Transformed Points , 2018, NeurIPS.

[15]  Benjamin Graham,et al.  Spatially-sparse convolutional neural networks , 2014, ArXiv.

[16]  Wei Wu,et al.  Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55 , 2017, ArXiv.

[17]  Bastian Leibe,et al.  Know What Your Neighbors Do: 3D Semantic Segmentation of Point Clouds , 2018, ECCV Workshops.

[18]  Ulrich Neumann,et al.  Recurrent Slice Networks for 3D Segmentation of Point Clouds , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Shichao Yang,et al.  Semantic 3D occupancy mapping through efficient high order CRFs , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[21]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Alexandre Boulch,et al.  SnapNet-R: Consistent 3D Multi-view Semantic Labeling for Robotics , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[23]  Duc Thanh Nguyen,et al.  JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Jiamao Li,et al.  3D Recurrent Neural Networks with Context Fusion for Point Cloud Semantic Segmentation , 2018, ECCV.

[25]  Alexandre Boulch,et al.  Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks , 2017, 3DOR@Eurographics.

[26]  Jing Huang,et al.  Pole-like object detection and classification from urban point clouds , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[29]  Ulrich Neumann,et al.  Exemplar-Based 3D Shape Segmentation in Point Clouds , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[30]  Iasonas Kokkinos,et al.  Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs , 2014, ICLR.

[31]  Ben Graham,et al.  Sparse 3D convolutional neural networks , 2015, BMVC.

[32]  Dushyant Rao,et al.  Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Kurt Keutzer,et al.  SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Jing Huang,et al.  Point cloud matching based on 3D self-similarity , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[35]  Ulrich Neumann,et al.  Pipe-Run Extraction and Reconstruction from Point Clouds , 2014, ECCV.

[36]  Fuxin Li,et al.  PointConv: Deep Convolutional Networks on 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Silvio Savarese,et al.  SEGCloud: Semantic Segmentation of 3D Point Clouds , 2017, 2017 International Conference on 3D Vision (3DV).

[38]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[39]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[40]  Duc Thanh Nguyen,et al.  Real-Time Progressive 3D Semantic Segmentation for Indoor Scenes , 2018, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[41]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Ali O. Ulusoy,et al.  Supplementary Material for OctNet : Learning Deep 3 D Representations at High Resolutions , 2017 .

[43]  Victor S. Lempitsky,et al.  Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Martin Simonovsky,et al.  Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Suya You,et al.  Estimation of camera pose with respect to terrestrial LiDAR data , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[46]  Leonidas J. Guibas,et al.  FPNN: Field Probing Neural Networks for 3D Data , 2016, NIPS.

[47]  Markus Vincze,et al.  Fast semantic segmentation of 3D point clouds using a dense CRF with learned parameters , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[48]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[50]  Cewu Lu,et al.  PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic Segmentation , 2018, ArXiv.

[51]  Bastian Leibe,et al.  Dense 3D semantic mapping of indoor scenes from RGB-D images , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).