EC-Net: an Edge-aware Point set Consolidation Network

Point clouds obtained from 3D scans are typically sparse, irregular, and noisy, and required to be consolidated. In this paper, we present the first deep learning based edge-aware technique to facilitate the consolidation of point clouds. We design our network to process points grouped in local patches, and train it to learn and help consolidate points, deliberately for edges. To achieve this, we formulate a regression component to simultaneously recover 3D point coordinates and point-to-edge distances from upsampled features, and an edge-aware joint loss function to directly minimize distances from output points to 3D meshes and to edges. Compared with previous neural network based works, our consolidation is edge-aware. During the synthesis, our network can attend to the detected sharp edges and enable more accurate 3D reconstructions. Also, we trained our network on virtual scanned point clouds, demonstrated the performance of our method on both synthetic and real point clouds, presented various surface reconstruction results, and showed how our method outperforms the state-of-the-arts.

[1]  H. Tal-Ezer,et al.  Parameterization-free projection for geometry reconstruction , 2007, SIGGRAPH 2007.

[2]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[3]  Mathieu Aubry,et al.  AtlasNet: A Papier-M\^ach\'e Approach to Learning 3D Surface Generation , 2018, CVPR 2018.

[4]  Matthias Nießner,et al.  Shape Completion Using 3D-Encoder-Predictor CNNs and Shape Synthesis , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Marshall W. Bern,et al.  A new Voronoi-based surface reconstruction algorithm , 1998, SIGGRAPH.

[6]  Yang Liu,et al.  O-CNN , 2017, ACM Trans. Graph..

[7]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Mathieu Aubry,et al.  A Papier-Mache Approach to Learning 3D Surface Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[10]  Daniel Cohen-Or,et al.  Consolidation of unorganized point clouds for surface reconstruction , 2009, ACM Trans. Graph..

[11]  Michael Wimmer,et al.  Continuous projection for fast L1 reconstruction , 2014, ACM Trans. Graph..

[12]  Binh-Son Hua,et al.  Point-wise Convolutional Neural Network , 2017, ArXiv.

[13]  Matthias Zwicker,et al.  GPF: GMM-Inspired Feature-Preserving Point Set Filtering , 2018, IEEE Transactions on Visualization and Computer Graphics.

[14]  Markus H. Gross,et al.  Shape modeling with point-sampled geometry , 2003, ACM Trans. Graph..

[15]  Daniel Cohen-Or,et al.  Surface reconstruction using local shape priors , 2007, Symposium on Geometry Processing.

[16]  Victor S. Lempitsky,et al.  Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[17]  Pierre Alliez,et al.  A Survey of Surface Reconstruction from Point Clouds , 2017, Comput. Graph. Forum.

[18]  Martin Simonovsky,et al.  Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Vladimir G. Kim,et al.  Data‐Driven Shape Analysis and Processing , 2015, Comput. Graph. Forum.

[20]  Youyi Zheng,et al.  IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 1 Bilateral Normal Filtering for Mesh Denoising , 2022 .

[21]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[23]  Binh-Son Hua,et al.  Pointwise Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[25]  Maks Ovsjanikov,et al.  PCPNet Learning Local Shape Properties from Raw Point Clouds , 2017, Comput. Graph. Forum.

[26]  Daniel Cohen-Or,et al.  PU-Net: Point Cloud Upsampling Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Qian Xie,et al.  Surface reconstruction with data-driven exemplar priors , 2017, Comput. Aided Des..

[28]  Xiaowu Chen,et al.  3D Mesh Labeling via Deep Convolutional Neural Networks , 2015, ACM Trans. Graph..

[29]  Tony DeRose,et al.  Surface reconstruction from unorganized points , 1992, SIGGRAPH.

[30]  David Eberly,et al.  Distance Between Point and Triangle in 3D , 2008 .

[31]  Leonidas J. Guibas,et al.  Data-driven structural priors for shape completion , 2015, ACM Trans. Graph..

[32]  Ulrich Neumann,et al.  SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Markus H. Gross,et al.  Feature Preserving Point Set Surfaces based on Non‐Linear Kernel Regression , 2009, Comput. Graph. Forum.

[34]  Gabriel Taubin,et al.  A benchmark for surface reconstruction , 2013, TOGS.

[35]  Leonidas J. Guibas,et al.  Frustum PointNets for 3D Object Detection from RGB-D Data , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[36]  Alexandre Boulch,et al.  Deep Learning for Robust Normal Estimation in Unstructured Point Clouds , 2016, Comput. Graph. Forum.

[37]  Daniel Cohen-Or,et al.  Edge-aware point set resampling , 2013, ACM Trans. Graph..

[38]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Baoquan Chen,et al.  PointCNN , 2018, NIPS 2018.

[40]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[41]  Loïc Barthe,et al.  Real-Time Point Cloud Refinement , 2004, PBG.

[42]  Gabriel Taubin,et al.  The ball-pivoting algorithm for surface reconstruction , 1999, IEEE Transactions on Visualization and Computer Graphics.

[43]  Zhen Li,et al.  High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[44]  Thomas Brox,et al.  Global, Dense Multiscale Reconstruction for a Billion Points , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[45]  Jiwen Lu,et al.  3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-Scale 3D Point Clouds , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[46]  Subhransu Maji,et al.  SPLATNet: Sparse Lattice Networks for Point Cloud Processing , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[47]  Marc Alexa,et al.  Computing and Rendering Point Set Surfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[48]  Hans-Peter Seidel,et al.  Design and volume optimization of space structures , 2017, ACM Trans. Graph..

[49]  Dong Tian,et al.  FoldingNet: Interpretable Unsupervised Learning on 3D Point Clouds , 2017, ArXiv.

[50]  Danfei Xu,et al.  PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[51]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[52]  D. Cohen-Or,et al.  Robust moving least-squares fitting with sharp features , 2005, ACM Trans. Graph..

[53]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Chen Kong,et al.  Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction , 2017, AAAI.