CGNet: A Cascaded Generative Network for dense point cloud reconstruction from a single image

Abstract Point cloud reconstruction has made great progress with the application of deep learning, but the blurred edges and sparse distribution of point clouds remain huge challenges in this field. In this paper, we propose a Cascaded Generative Network (CGNet) to reconstruct dense point clouds from a single image. To preserve shape features, the pre-reconstruction network is combined with the up-sampling network to construct the multi-stage generation framework. In the generation process, an image re-description mechanism is designed to supervise the entire network by regenerating images from the reconstructed point clouds. Furthermore, the generative network introduces a siamese structure to extract consistent high-level semantic from multiple images. Extensive experiments on the ShapeNet dataset demonstrate that CGNet outperforms the state-of-the-art point cloud reconstruction methods.

[1]  Honglak Lee,et al.  Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision , 2016, NIPS.

[2]  Jiajun Wu,et al.  Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[3]  Jiande Sun,et al.  High-Resolution Point Cloud Reconstruction from a Single Image by Redescription , 2020, 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[4]  Jitendra Malik,et al.  Learning a Multi-View Stereo Machine , 2017, NIPS.

[5]  Daniel Cohen-Or,et al.  Edge-aware point set resampling , 2013, ACM Trans. Graph..

[6]  Martial Hebert,et al.  PCN: Point Completion Network , 2018, 2018 International Conference on 3D Vision (3DV).

[7]  Silvio Savarese,et al.  Weakly Supervised 3D Reconstruction with Adversarial Constraint , 2017, 2017 International Conference on 3D Vision (3DV).

[8]  Yi Yang,et al.  PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing , 2019, ArXiv.

[9]  Abhinav Gupta,et al.  Learning a Predictable and Generative Vector Representation for Objects , 2016, ECCV.

[10]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  R. Venkatesh Babu,et al.  CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision , 2018, AAAI.

[12]  Hans-Peter Seidel,et al.  Design and volume optimization of space structures , 2017, ACM Trans. Graph..

[13]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[14]  Nahum Kiryati,et al.  Photometric stereo under perspective projection , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[15]  Andrew Blake,et al.  Shape from texture: the homogeneity hypothesis , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[16]  Chen Kong,et al.  Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction , 2017, AAAI.

[17]  Alexei A. Efros,et al.  Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics , 2010, ECCV.

[18]  Daniel Cohen-Or,et al.  PU-Net: Point Cloud Upsampling Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Tatsuya Harada,et al.  Neural 3D Mesh Renderer , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[20]  J. Tenenbaum,et al.  MarrNet : 3 D Shape Reconstruction via 2 . 5 D Sketches , 2017 .

[21]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[22]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[23]  David A. Forsyth,et al.  Generalizing motion edits with Gaussian processes , 2009, ACM Trans. Graph..

[24]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[25]  Anders P. Eriksson,et al.  Image2Mesh: A Learning Framework for Single Image 3D Reconstruction , 2017, ACCV.

[26]  Leonidas J. Guibas,et al.  Representation Learning and Adversarial Generation of 3D Point Clouds , 2017, ArXiv.

[27]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[28]  Kaiqi Huang,et al.  Point Cloud Super Resolution with Adversarial Residual Graph Networks , 2019, BMVC.

[29]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[30]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[31]  Jitendra Malik,et al.  Category-specific object reconstruction from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  U. Ascher,et al.  Consolidation of unorganized point clouds for surface reconstruction , 2009, ACM Trans. Graph..

[33]  R. Venkatesh Babu,et al.  3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image , 2018, BMVC.

[34]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Daniel Cohen-Or,et al.  Patch-Based Progressive 3D Point Set Upsampling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Guillermo Sapiro,et al.  Seeing 3D objects in a single 2D image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[37]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Hans-Peter Seidel,et al.  An efficient construction of reduced deformable objects , 2013, ACM Trans. Graph..

[40]  R. Venkatesh Babu,et al.  Dense 3D Point Cloud Reconstruction Using a Deep Pyramid Network , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).