Cloth and Skin Deformation with a Triangle Mesh Based Convolutional Neural Network

We introduce a triangle mesh based convolutional neural network. The proposed network structure can be used for problems where input and/or output are defined on a manifold triangle mesh with or without boundary. We demonstrate its applications in cloth upsampling, adding back details to Principal Component Analysis (PCA) compressed cloth, regressing clothing deformation from character poses, and regressing hand skin deformation from bones' joint angles. The data used for training in this work are generated from high resolution extended position based dynamics (XPBD) physics simulations with small time steps and high iteration counts and from an offline FEM simulator, but it can come from other sources. The inference time of our prototype implementation, depending on the mesh resolution and the network size, can provide between 4 to 134 times faster than a GPU based simulator. The inference also only needs to be done for meshes currently visible by the camera.

[1]  Derek Nowrouzezahrai,et al.  Subspace neural physics: fast data-driven interactive simulation , 2019, Symposium on Computer Animation.

[2]  Matthias Harders,et al.  Enriching coarse interactive elastic objects with high-resolution data-driven deformations , 2012, SCA '12.

[3]  Nadia Magnenat-Thalmann,et al.  Animating wrinkles on clothes , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[4]  Daniel Cohen-Or,et al.  MeshCNN: a network with an edge , 2019, ACM Trans. Graph..

[5]  Maks Ovsjanikov,et al.  Multi-directional geodesic neural networks via equivariant convolution , 2018, ACM Trans. Graph..

[6]  Daniel Cremers,et al.  DeepWrinkles: Accurate and Realistic Clothing Modeling , 2018, ECCV.

[7]  Byung-Uck Kim,et al.  A deformation transformer for real-time cloth animation , 2010, ACM Trans. Graph..

[8]  Jaakko Lehtinen,et al.  Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer , 2019, NeurIPS.

[9]  Ronald Fedkiw,et al.  A Pixel‐Based Framework for Data‐Driven Clothing , 2018, Comput. Graph. Forum.

[10]  Bruno Lévy,et al.  Mesh parameterization: theory and practice , 2007, SIGGRAPH Courses.

[11]  Peter-Pike J. Sloan,et al.  Physics-inspired upsampling for cloth simulation in games , 2011, ACM Trans. Graph..

[12]  Stefanos Zafeiriou,et al.  SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[13]  Tomas E. Ward,et al.  Generative Adversarial Networks: A Survey and Taxonomy , 2019, ArXiv.

[14]  Marie-Paule Cani,et al.  Real-time dynamic wrinkles , 2004, Proceedings Computer Graphics International, 2004..

[15]  Marcel Campen,et al.  A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3D Meshes , 2018, ECCV Workshops.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  James F. O'Brien,et al.  Near-exhaustive precomputation of secondary cloth effects , 2013, ACM Trans. Graph..

[18]  Lan Chen,et al.  Synthesizing cloth wrinkles by CNN‐based geometry image superresolution , 2018, Comput. Animat. Virtual Worlds.

[19]  Miguel A. Otaduy,et al.  Learning‐Based Animation of Clothing for Virtual Try‐On , 2019, Comput. Graph. Forum.

[20]  Jonathan Masci,et al.  Learning shape correspondence with anisotropic convolutional neural networks , 2016, NIPS.

[21]  Pierre Vandergheynst,et al.  Geodesic Convolutional Neural Networks on Riemannian Manifolds , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[22]  Jonathan Masci,et al.  Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[24]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Matthias Müller,et al.  Wrinkle meshes , 2010, SCA '10.

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  Adam W. Bargteil,et al.  Physics-inspired upsampling for cloth simulation in games , 2011, SIGGRAPH 2011.

[28]  Byung-Uck Kim,et al.  A deformation transformer for real-time cloth animation , 2010, SIGGRAPH 2010.

[29]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[30]  H. Kuhn.  The Hungarian method for the assignment problem , 1955 .

[31]  Vladimir G. Kim,et al.  GWCNN: A Metric Alignment Layer for Deep Shape Analysis , 2017, Comput. Graph. Forum.

[32]  Jeongjin Lee,et al.  A real-time cloth draping simulation algorithm using conjugate harmonic functions , 2007, Comput. Graph..

[33]  Ersin Yumer,et al.  Convolutional neural networks on surfaces via seamless toric covers , 2017, ACM Trans. Graph..

[34]  Matthias Müller,et al.  XPBD: position-based simulation of compliant constrained dynamics , 2016, MIG.

[35]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[36]  In-Kwon Lee,et al.  Efficient Cloth Simulation using Miniature Cloth and Upscaling Deep Neural Networks , 2019, ArXiv.

[37]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[38]  In-Kwon Lee,et al.  Hierarchical Cloth Simulation using Deep Neural Networks , 2018, CGI 2018.

[39]  Lin Gao,et al.  Realtime Simulation of Thin-Shell Deformable Materials Using CNN-Based Mesh Embedding , 2020, IEEE Robotics and Automation Letters.

[40]  A. Volgenant,et al.  A shortest augmenting path algorithm for dense and sparse linear assignment problems , 1987, Computing.

[41]  Hwan-Gue Cho,et al.  An efficient animation of wrinkled cloth with approximate implicit integration , 2001, The Visual Computer.

[42]  Yaron Lipman,et al.  Surface Networks via General Covers , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[43]  Jorge Nocedal,et al.  Representations of quasi-Newton matrices and their use in limited memory methods , 1994, Math. Program..

[44]  Alla Sheffer,et al.  Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles , 2010, ACM Trans. Graph..

[45]  James F. O'Brien,et al.  Fast and deep deformation approximations , 2018, ACM Trans. Graph..

[46]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[47]  Paul J. Zsombor-Murray,et al.  Direct and specific least-square fitting of hyperbolæ and ellipses , 2004, J. Electronic Imaging.

[48]  Ronald Fedkiw,et al.  Inequality cloth , 2017, Symposium on Computer Animation.

[49]  David I. W. Levin,et al.  Latent‐space Dynamics for Reduced Deformable Simulation , 2019, Comput. Graph. Forum.

[50]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[51]  Stefanos Zafeiriou,et al.  Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).