Mixed Feature Prediction on Boundary Learning for Point Cloud Semantic Segmentation

Existing point cloud semantic segmentation approaches do not perform well on details, especially for the boundary regions. However, supervised-learning-based methods depend on costly artificial annotations for performance improvement. In this paper, we bridge this gap by designing a self-supervised pretext task applicable to point clouds. Our main innovation lies in the mixed feature prediction strategy during the pretraining stage, which facilitates point cloud feature learning with boundary-aware foundations. Meanwhile, a dynamic feature aggregation module is proposed to regulate the range of receptive field according to the neighboring pattern of each point. In this way, more spatial details are preserved for discriminative high-level representations. Extensive experiments across several point cloud segmentation datasets verify the superiority of our proposed method, including ShapeNet-part, ScanNet v2, and S3DIS. Furthermore, transfer learning on point cloud classification and object detection tasks demonstrates the generalization ability of our method.

[1]  Yong Jae Lee,et al.  Masked Discrimination for Self-Supervised Learning on Point Clouds , 2022, ECCV.

[2]  D. Tao,et al.  Contrastive Boundary Learning for Point Cloud Segmentation , 2022, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Yingcai Wan,et al.  Multi-Sensor Fusion Self-Supervised Deep Odometry and Depth Estimation , 2022, Remote. Sens..

[4]  Guangsheng Chen,et al.  AGNet: An Attention-Based Graph Network for Point Cloud Classification and Segmentation , 2022, Remote. Sens..

[5]  Y. Fu,et al.  Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework , 2022, ICLR.

[6]  J. Stoter,et al.  City3D: Large-Scale Building Reconstruction from Airborne LiDAR Point Clouds , 2022, Remote. Sens..

[7]  A. Yuille,et al.  Masked Feature Prediction for Self-Supervised Visual Pre-Training , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jiwen Lu,et al.  Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Han Hu,et al.  SimMIM: a Simple Framework for Masked Image Modeling , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Ross B. Girshick,et al.  Masked Autoencoders Are Scalable Vision Learners , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jihong Zhu,et al.  PointCutMix: Regularization Strategy for Point Cloud Classification , 2021, Neurocomputing.

[12]  Naoki Suganuma,et al.  Graph SLAM-Based 2.5D LIDAR Mapping Module for Autonomous Vehicles , 2021, Remote. Sens..

[13]  Alexey Nekrasov,et al.  Mix3D: Out-of-Context Data Augmentation for 3D Scenes , 2021, 2021 International Conference on 3D Vision (3DV).

[14]  Yongyang Xu,et al.  DGANet: A Dilated Graph Attention-Based Network for Local Feature Extraction on 3D Point Clouds , 2021, Remote. Sens..

[15]  Jiwen Lu,et al.  PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Jan Kautz,et al.  Self-Supervised Learning on 3D Point Clouds by Learning Discrete Generative Models , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yisheng Lv,et al.  SCF-Net: Learning Spatial Contextual Features for Large-Scale Point Cloud Segmentation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Weidong Cai,et al.  Walk in the Cloud: Learning Curves for Point Clouds Shape Analysis , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[19]  Zheng Zhang,et al.  Group-Free 3D Object Detection via Transformers , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  N. Barnes,et al.  Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Songyang Zhang,et al.  Point Cloud Resampling via Hypergraph Signal Processing , 2021, IEEE Signal Processing Letters.

[22]  Bisheng Yang,et al.  Autonomous Vehicle Localization with Prior Visual Point Cloud Map Constraints in GNSS-Challenged Environments , 2021, Remote. Sens..

[23]  François Jonard,et al.  Estimating Forest Structure from UAV-Mounted LiDAR Point Cloud Using Machine Learning , 2021, Remote. Sens..

[24]  Rohit Girdhar,et al.  Self-Supervised Pretraining of 3D Features on any Point-Cloud , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Ralph R. Martin,et al.  PCT: Point cloud transformer , 2020, Computational Visual Media.

[26]  Klaus Dietmayer,et al.  Point Transformer , 2020, IEEE Access.

[27]  Matt J. Kusner,et al.  Unsupervised Point Cloud Pre-training via Occlusion Completion , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  A. Mian,et al.  Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Trapit Bansal,et al.  Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks , 2020, EMNLP.

[30]  Leonidas J. Guibas,et al.  PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding , 2020, ECCV.

[31]  Jianping Shi,et al.  Improving Semantic Segmentation via Decoupled Body and Edge Supervision , 2020, ECCV.

[32]  Hongbo Fu,et al.  JSENet: Joint Semantic Segmentation and Edge Detection Network for 3D Point Clouds , 2020, ECCV.

[33]  Haitao Yang,et al.  H3DNet: 3D Object Detection Using Hybrid Geometric Primitives , 2020, ECCV.

[34]  Jinglu Wang,et al.  Joint Semantic Segmentation and Boundary Detection Using Iterative Pyramid Contexts , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Yang Li,et al.  Geometry-Driven Self-Supervised Method for 3D Human Pose Estimation , 2020, AAAI.

[36]  Jan Kautz,et al.  Self-supervised Single-view 3D Reconstruction via Semantic Consistency , 2020, ECCV.

[37]  Leonidas J. Guibas,et al.  ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  A. Markham,et al.  RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Guo-Jun Qi,et al.  GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-Wise Transformations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Chi-Wing Fu,et al.  Hierarchical Point-Edge Interaction Network for Point Cloud Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Duc Thanh Nguyen,et al.  Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[42]  Leonidas J. Guibas,et al.  Deep Hough Voting for 3D Object Detection in Point Clouds , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  Silvio Savarese,et al.  4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Leonidas J. Guibas,et al.  KPConv: Flexible and Deformable Convolution for Point Clouds , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[45]  Shiming Xiang,et al.  Relation-Shape Convolutional Neural Network for Point Cloud Analysis , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Jonathan Sauder,et al.  Self-Supervised Deep Learning on Point Clouds by Reconstructing Space , 2019, NeurIPS.

[47]  Federico Tombari,et al.  3D Point Capsule Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Fuxin Li,et al.  PointConv: Deep Convolutional Networks on 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Yang Zhang,et al.  Point Cloud GAN , 2018, DGS@ICLR.

[50]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[51]  Frank Hutter,et al.  Decoupled Weight Decay Regularization , 2017, ICLR.

[52]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[53]  Wei Wu,et al.  PointCNN: Convolution On X-Transformed Points , 2018, NeurIPS.

[54]  Daniel Cohen-Or,et al.  EC-Net: an Edge-aware Point set Consolidation Network , 2018, ECCV.

[55]  Subhransu Maji,et al.  Multiresolution Tree Networks for 3D Point Cloud Processing , 2018, ECCV.

[56]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[57]  Martin Simonovsky,et al.  Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[58]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[59]  Silvio Savarese,et al.  SEGCloud: Semantic Segmentation of 3D Point Clouds , 2017, 2017 International Conference on 3D Vision (3DV).

[60]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[61]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[63]  Peng-Shuai Wang,et al.  O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis , 2017, ArXiv.

[64]  Leonidas J. Guibas,et al.  A scalable active framework for region annotation in 3D shape collections , 2016, ACM Trans. Graph..

[65]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[66]  Silvio Savarese,et al.  3D Semantic Parsing of Large-Scale Indoor Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Paolo Favaro,et al.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles , 2016, ECCV.

[68]  C. Qi Deep Learning on Point Sets for 3 D Classification and Segmentation , 2016 .

[69]  Markus Vincze,et al.  Fast and accurate normal estimation by efficient 3d edge detection , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[70]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[71]  Javier Ruiz Hidalgo,et al.  Fast and Robust Edge Extraction in Unorganized Point Clouds , 2015, 2015 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[72]  Lei Guo,et al.  Weakly Supervised Learning for Target Detection in Remote Sensing Images , 2015, IEEE Geoscience and Remote Sensing Letters.

[73]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  Bolei Zhou,et al.  Learning Deep Features for Scene Recognition using Places Database , 2014, NIPS.

[75]  Eric L. Miller,et al.  Three-Dimensional Surface Mesh Segmentation Using Curvedness-Based Region Growing Approach , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  T. Rabbani,et al.  SEGMENTATION OF POINT CLOUDS USING SMOOTHNESS CONSTRAINT , 2006 .

[77]  G. Sithole,et al.  Recognising structure in laser scanning point clouds , 2004 .

[78]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.