Temporally-Consistent Surface Reconstruction using Metrically-Consistent Atlases

We propose a method for unsupervised reconstruction of a temporally-consistent sequence of surfaces from a sequence of time-evolving point clouds. It yields dense and semantically meaningful correspondences between frames. We represent the reconstructed surfaces as atlases computed by a neural network, which enables us to establish correspondences between frames. The key to making these correspondences semantically meaningful is to guarantee that the metric tensors computed at corresponding points are as similar as possible. We have devised an optimization strategy that makes our method robust to noise and global motions, without a priori correspondences or pre-alignment steps. As a result, our approach outperforms state-of-the-art ones on several challenging datasets. The code is available at https://github.com/bednarikjan/temporally_coherent_surface_reconstruction.

[1]  Pascal Fua,et al.  Temporally-Coherent Surface Reconstruction via Metric-Consistent Atlases , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[2]  Dan Xu,et al.  Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Pascal Fua,et al.  Better Patch Stitching for Parametric Surface Reconstruction , 2020, 2020 International Conference on 3D Vision (3DV).

[4]  Maks Ovsjanikov,et al.  Intrinsic Point Cloud Interpolation via Dual Latent Space Navigation , 2020, ECCV.

[5]  Abhishek Sharma,et al.  Deep Geometric Functional Maps: Robust Feature Learning for Shape Correspondence , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  R. Kimmel,et al.  LIMP: Learning Latent Shape Representations with Metric Preservation Priors , 2020, ECCV.

[7]  Cewu Lu,et al.  KeypointNet: A Large-Scale 3D Keypoint Dataset Aggregated From Numerous Human Annotations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Pascal Fua,et al.  Shape Reconstruction by Learning Differentiable Surface Representations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Andreas Geiger,et al.  Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Xiaoguang Han,et al.  Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[11]  Mathieu Aubry,et al.  Learning elementary structures for 3D shape generation and matching , 2019, NeurIPS.

[12]  Michael J. Black,et al.  Learning to Dress 3D People in Generative Clothing , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Mathieu Aubry,et al.  Unsupervised cycle‐consistent deformation for shape matching , 2019, Comput. Graph. Forum.

[14]  Ron Kimmel,et al.  Unsupervised Learning of Dense Shape Correspondence , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Richard A. Newcombe,et al.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Maks Ovsjanikov,et al.  Unsupervised Deep Learning for Structured Shape Matching , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Maks Ovsjanikov,et al.  Isospectralization, or How to Hear Shape, Style, and Correspondence , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Joan Bruna,et al.  Deep Geometric Prior for Surface Reconstruction , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Alexey Dosovitskiy,et al.  Unsupervised Learning of Shape and Pose with Differentiable Point Clouds , 2018, NeurIPS.

[21]  Keenan Crane,et al.  Möbius Registration , 2018, Comput. Graph. Forum.

[22]  Mathieu Aubry,et al.  3D-CODED: 3D Correspondences by Deep Deformation , 2018, ECCV.

[23]  Jitendra Malik,et al.  Learning Category-Specific Mesh Reconstruction from Image Collections , 2018, ECCV.

[24]  Mathieu Aubry,et al.  A Papier-Mache Approach to Learning 3D Surface Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Dong Tian,et al.  FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Lin Gao,et al.  Data‐Driven Shape Interpolation and Morphing Editing , 2017, Comput. Graph. Forum.

[27]  Michael J. Black,et al.  Dynamic FAUST: Registering Human Bodies in Motion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Ersin Yumer,et al.  Learning Local Shape Descriptors from Part Correspondences with Multiview Convolutional Networks , 2017, ACM Trans. Graph..

[29]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[30]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Jinlong Yang,et al.  Estimation of Human Body Shape in Motion with Wide Clothing , 2016, ECCV.

[33]  Huu Le,et al.  Conformal Surface Alignment with Optimal Möbius Search , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Takeo Kanade,et al.  Panoptic Studio: A Massively Multiview System for Social Motion Capture , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[35]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[36]  Michael J. Black,et al.  The stitched puppet: A graphical model of 3D human shape and pose , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Roi Poranne,et al.  Lifted bijections for low distortion surface mappings , 2014, ACM Trans. Graph..

[38]  Denis Zorin,et al.  Locally injective parametrization with arbitrary fixed boundaries , 2014, ACM Trans. Graph..

[39]  Daniel Cremers,et al.  Dense Non-rigid Shape Correspondence Using Random Forests , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Pierre Alliez,et al.  State of the Art in Surface Reconstruction from Point Clouds , 2014, Eurographics.

[41]  Patrice Koehl,et al.  Automatic Alignment of Genus-Zero Surfaces , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[43]  A. Hamza,et al.  A multiresolution descriptor for deformable 3D shape retrieval , 2013, The Visual Computer.

[44]  Daniel Cremers,et al.  The wave kernel signature: A quantum mechanical approach to shape analysis , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[45]  Edmond Boyer,et al.  Temporally Coherent Segmentation of 3D Reconstructions , 2010 .

[46]  K. Hormann,et al.  Multi‐Scale Geometry Interpolation , 2010, Comput. Graph. Forum.

[47]  I. Daubechies,et al.  Surface Comparison with Mass Transportation , 2009, 0912.3488.

[48]  Thomas A. Funkhouser,et al.  Möbius voting for surface correspondence , 2009, ACM Trans. Graph..

[49]  M. Ovsjanikov,et al.  A concise and provably informative multi-scale signature based on heat diffusion , 2009 .

[50]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[51]  R. Horaud,et al.  Coherent Laplacian 3-D protrusion segmentation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[52]  Ron Kimmel,et al.  Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[53]  Lok Ming Lui,et al.  Optimization of Brain Conformal Mapping with Landmarks , 2005, MICCAI.

[54]  Guillermo Sapiro,et al.  A Theoretical and Computational Framework for Isometry Invariant Recognition of Point Cloud Data , 2005, Found. Comput. Math..

[55]  Alla Sheffer,et al.  Cross-parameterization and compatible remeshing of 3D models , 2004, ACM Trans. Graph..

[56]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[57]  Marc Alexa,et al.  As-rigid-as-possible shape interpolation , 2000, SIGGRAPH.

[58]  David P. Dobkin,et al.  Multiresolution mesh morphing , 1999, SIGGRAPH.

[59]  Franck Hétroy,et al.  Segmentation of temporal mesh sequences into rigidly moving components , 2013, Graph. Model..

[60]  Pascal Fua,et al.  Model-Based Optimization: Accurate and Consistent Site Modeling , 1996 .

[61]  Dimitris N. Metaxas,et al.  Eurographics/ Acm Siggraph Symposium on Computer Animation (2007) Harmonic Skeleton for Realistic Character Animation , 2022 .

[62]  Daniel Cohen-Or,et al.  Eurographics Symposium on Geometry Processing (2007) Data-dependent Mls for Faithful Surface Approximation , 2022 .