DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces

Shape deformation is an important component in any geometry processing toolbox. The goal is to enable intuitive deformations of single or multiple shapes or to transfer example deformations to new shapes while preserving the plausibility of the deformed shape(s). Existing approaches assume access to point-level or part-level correspondence or establish them in a preprocessing phase, thus limiting the scope and generality of such approaches. We propose DeformSyncNet, a new approach that allows consistent and synchronized shape deformations without requiring explicit correspondence information. Technically, we achieve this by encoding deformations into a class-specific idealized latent space while decoding them into an individual, model-specific linear deformation action space, operating directly in 3D. The underlying encoding and decoding are performed by specialized (jointly trained) neural networks. By design, the inductive bias of our networks results in a deformation space with several desirable properties, such as path invariance across different deformation pathways, which are then also approximately preserved in real space. We qualitatively and quantitatively evaluate our framework against multiple alternative approaches and demonstrate improved performance.

[1]  Leonidas J. Guibas,et al.  A scalable active framework for region annotation in 3D shape collections , 2016, ACM Trans. Graph..

[2]  Olga Sorkine-Hornung,et al.  Neural Cages for Detail-Preserving 3D Deformations , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[4]  Leonidas J. Guibas,et al.  ComplementMe , 2017, ACM Trans. Graph..

[5]  Jessica K. Hodgins,et al.  Realtime style transfer for unlabeled heterogeneous human motion , 2015, ACM Trans. Graph..

[6]  M. V. D. Panne,et al.  Joint-aware manipulation of deformable models , 2009, SIGGRAPH 2009.

[7]  Leonidas J. Guibas,et al.  Learning Shape Abstractions by Assembling Volumetric Primitives , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[9]  Silvio Savarese,et al.  DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[10]  Jovan Popović,et al.  Semantic deformation transfer , 2009, SIGGRAPH 2009.

[11]  Alla Sheffer,et al.  Analogy‐driven 3D style transfer , 2014, Comput. Graph. Forum.

[12]  Daniel Cohen-Or,et al.  ALIGNet: Partial-Shape Agnostic Alignment via Unsupervised Learning , 2018, ACM Trans. Graph..

[13]  D. Levin,et al.  Linear rotation-invariant coordinates for meshes , 2005, SIGGRAPH 2005.

[14]  Radomír Mech,et al.  3DN: 3D Deformation Network , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Thomas A. Funkhouser,et al.  Learning Shape Templates With Structured Implicit Functions , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  Frédéric Maire,et al.  Learning Free-Form Deformations for 3D Object Reconstruction , 2018, ACCV.

[17]  Xiaogang Wang,et al.  Shape2Motion: Joint Analysis of Motion Parts and Attributes From 3D Shapes , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[19]  Leonidas J. Guibas,et al.  Limit Shapes – A Tool for Understanding Shape Differences and Variability in 3D Model Collections , 2019, Comput. Graph. Forum.

[20]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Ruben Villegas,et al.  Neural Kinematic Networks for Unsupervised Motion Retargetting , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Lin Gao,et al.  Automatic unpaired shape deformation transfer , 2018, ACM Trans. Graph..

[23]  Leonidas J. Guibas,et al.  Map-based exploration of intrinsic shape differences and variability , 2013, ACM Trans. Graph..

[24]  Mathieu Aubry,et al.  AtlasNet: A Papier-M\^ach\'e Approach to Learning 3D Surface Generation , 2018, CVPR 2018.

[25]  Leonidas J. Guibas,et al.  Deep Functional Dictionaries: Learning Consistent Semantic Structures on 3D Models from Functions , 2018, NeurIPS.

[26]  Thomas A. Funkhouser,et al.  Interactive 3D Modeling with a Generative Adversarial Network , 2017, 2017 International Conference on 3D Vision (3DV).

[27]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[29]  Daniel Cohen-Or,et al.  Component‐wise Controllers for Structure‐Preserving Shape Manipulation , 2011, Comput. Graph. Forum.

[30]  Lin Gao,et al.  Biharmonic deformation transfer with automatic key point selection , 2018, Graph. Model..

[31]  Leonidas J. Guibas,et al.  OperatorNet: Recovering 3D Shapes From Difference Operators , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Leonidas J. Guibas,et al.  StructEdit: Learning Structural Shape Variations , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Leonidas J. Guibas,et al.  Supervised Fitting of Geometric Primitives to 3D Point Clouds , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[35]  Hujun Bao,et al.  Cage-based deformation transfer , 2010, Comput. Graph..

[36]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[37]  Craig Gotsman,et al.  Spatial deformation transfer , 2009, SCA '09.

[38]  Christian Rössl,et al.  Laplacian surface editing , 2004, SGP '04.

[39]  Matthieu Cord,et al.  DiscoNet: Shapes Learning on Disconnected Manifolds for 3D Editing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[40]  Kun Zhou,et al.  Deformation Transfer to Multi‐Component Objects , 2010, Comput. Graph. Forum.

[41]  Jun Li,et al.  Symmetry Hierarchy of Man‐Made Objects , 2011, Comput. Graph. Forum.

[42]  Levent Burak Kara,et al.  Co-constrained handles for deformation in shape collections , 2014, ACM Trans. Graph..

[43]  Leonidas J. Guibas,et al.  Shapeglot: Learning Language for Shape Differentiation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[44]  Agustí Reventós Tarrida Affine Maps, Euclidean Motions and Quadrics , 2011 .

[45]  Leonidas J. Guibas,et al.  Learning Representations and Generative Models for 3D Point Clouds , 2017, ICML.

[46]  Jean Gallier,et al.  Geometric Methods and Applications , 2011 .

[47]  Ersin Yumer,et al.  Learning Local Shape Descriptors from Part Correspondences with Multiview Convolutional Networks , 2017, ACM Trans. Graph..

[48]  Mathieu Aubry,et al.  A Papier-Mache Approach to Learning 3D Surface Generation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[49]  N. Mitra,et al.  iWIRES: an analyze-and-edit approach to shape manipulation , 2009, SIGGRAPH 2009.

[50]  Daniel Cohen-Or,et al.  Recurring part arrangements in shape collections , 2014, Comput. Graph. Forum.

[51]  Chris H. Q. Ding,et al.  R1-PCA: rotational invariant L1-norm principal component analysis for robust subspace factorization , 2006, ICML.

[52]  Stephen DiVerdi,et al.  Learning part-based templates from large collections of 3D shapes , 2013, ACM Trans. Graph..

[53]  Leonidas J. Guibas,et al.  Exploration of continuous variability in collections of 3D shapes , 2011, ACM Trans. Graph..

[54]  Leonidas J. Guibas,et al.  Learning hierarchical shape segmentation and labeling from online repositories , 2017, ACM Trans. Graph..

[55]  D. Cohen-Or,et al.  Style-content separation by anisotropic part scales , 2010, ACM Trans. Graph..

[56]  Takeo Igarashi,et al.  As-rigid-as-possible shape manipulation , 2005, SIGGRAPH '05.

[57]  Hao Li,et al.  Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans , 2008, Comput. Graph. Forum.

[58]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[59]  Q. Mcnemar Note on the sampling error of the difference between correlated proportions or percentages , 1947, Psychometrika.

[60]  Wojciech Matusik,et al.  Retrieval on Parametric Shape Collections , 2017, ACM Trans. Graph..

[61]  Daniel Cohen-Or,et al.  Meta-representation of shape families , 2014, ACM Trans. Graph..

[62]  Niloy J. Mitra,et al.  Learning Semantic Deformation Flows with 3D Convolutional Networks , 2016, ECCV.

[63]  Jiajun Wu,et al.  Learning to Infer and Execute 3D Shape Programs , 2019, ICLR.

[64]  Daniel Cohen-Or,et al.  iWIRES: an analyze-and-edit approach to shape manipulation , 2009, ACM Trans. Graph..

[65]  Federico Tombari,et al.  3D Point Capsule Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Leonidas J. Guibas,et al.  PartNet: A Large-Scale Benchmark for Fine-Grained and Hierarchical Part-Level 3D Object Understanding , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[68]  Hao Su,et al.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[69]  Leonidas J. Guibas,et al.  StructureNet , 2019, ACM Trans. Graph..

[70]  Daniel Cohen-Or,et al.  LOGAN , 2019, ACM Trans. Graph..