Physics-Inspired Garment Recovery from a Single-View Image

Most recent garment capturing techniques rely on acquiring multiple views of clothing, which may not always be readily available, especially in the case of pre-existing photographs from the web. As an alternative, we propose a method that is able to compute a 3D model of a human body and its outfit from a single photograph with little human interaction. Our algorithm is not only able to capture the global shape and overall geometry of the clothing, it can also extract the physical properties (i.e., material parameters needed for simulation) of cloth. Unlike previous methods using full 3D information (i.e., depth, multi-view images, or sampled 3D geometry), our approach achieves garment recovery from a single-view image by using physical, statistical, and geometric priors and a combination of parameter estimation, semantic parsing, shape/pose recovery, and physics-based cloth simulation. We demonstrate the effectiveness of our algorithm by re-purposing the reconstructed garments for virtual try-on and garment transfer applications and for cloth animation on digital characters.

[1]  Huamin Wang,et al.  Real-time Human Pose and Shape Estimation for Virtual Try-On Using a Single Commodity Depth Camera , 2014, IEEE Trans. Vis. Comput. Graph..

[2]  Azriel Rosenfeld,et al.  Face recognition: A literature survey , 2003, CSUR.

[3]  David A. Forsyth,et al.  Capturing and animating occluded cloth , 2007, ACM Trans. Graph..

[4]  Alexander M. Bronstein,et al.  Numerical Geometry of Non-Rigid Shapes , 2009, Monographs in Computer Science.

[5]  Alla Sheffer,et al.  Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles , 2010, ACM Trans. Graph..

[6]  Eitan Grinspun,et al.  Parsing sewing patterns into 3D garments , 2013, ACM Trans. Graph..

[7]  Alexander M. Bronstein,et al.  Efficient Computation of Isometry-Invariant Distances Between Surfaces , 2006, SIAM J. Sci. Comput..

[8]  Alla Sheffer,et al.  Context-aware garment modeling from sketches , 2011, Comput. Graph..

[9]  Dinesh Manocha,et al.  ICCD: Interactive Continuous Collision Detection between Deformable Models Using Connectivity-Based Culling , 2008, IEEE Transactions on Visualization and Computer Graphics.

[10]  David E. Breen,et al.  Cloth modeling and animation , 2000 .

[11]  James F. O'Brien,et al.  Adaptive anisotropic remeshing for cloth simulation , 2012, ACM Trans. Graph..

[12]  Andrew P. Witkin,et al.  Large steps in cloth simulation , 1998, SIGGRAPH.

[13]  Vladimir G. Kim,et al.  Physics-driven pattern adjustment for direct 3D garment editing , 2016, ACM Trans. Graph..

[14]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[15]  Kun Zhou,et al.  3D shape regression for real-time facial animation , 2013, ACM Trans. Graph..

[16]  Takeo Igarashi,et al.  As-rigid-as-possible shape manipulation , 2005, SIGGRAPH '05.

[17]  Dinesh Manocha,et al.  ICCD: Interactive Continuous Collision Detection between Deformable Models Using Connectivity-Based Culling , 2009, IEEE Trans. Vis. Comput. Graph..

[18]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[19]  Volker Scholz,et al.  Texture replacement of garments in monocular video sequences , 2006, EGSR '06.

[20]  Yu Guo,et al.  Deformable model for estimating clothed and naked human shapes from a single image , 2013, The Visual Computer.

[21]  Katsu Yamane,et al.  High Marker Density Motion Capture by Retroreflective Mesh Suit , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[22]  Bin Zhou,et al.  Garment Modeling from a Single Image , 2013, Comput. Graph. Forum.

[23]  Alla Sheffer,et al.  Author manuscript, published in "ACM Transactions on Graphics (2012)" DOI: 10.1145/2185520.2185532 Design Preserving Garment Transfer , 2012 .

[24]  Derek Bradley,et al.  High-quality capture of eyes , 2014, ACM Trans. Graph..

[25]  John F. Hughes,et al.  A Sketch-Based Interface for Clothing Virtual Characters , 2007, IEEE Computer Graphics and Applications.

[26]  Jessica K. Hodgins,et al.  A perceptual control space for garment simulation , 2015, ACM Trans. Graph..

[27]  Peter-Pike J. Sloan,et al.  Fast and Efficient Skinning of Animated Meshes , 2010, Comput. Graph. Forum.

[28]  Dinesh Manocha,et al.  Fast continuous collision detection among deformable models using graphics processors , 2006, EGVE'06.

[29]  Pascal Volino,et al.  Fast Geometrical Wrinkles on Animated Surfaces , 1998 .

[30]  D. Cohen-Or,et al.  Parametric reshaping of human bodies in images , 2010, ACM Trans. Graph..

[31]  Nadia Magnenat-Thalmann,et al.  An automatic modeling of human bodies from sizing parameters , 2003, I3D '03.

[32]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH '05.

[33]  J. Marks,et al.  Garment Motion Capture Using Color‐Coded Patterns , 2005, SIGGRAPH '05.

[34]  Hyeong-Seok Ko,et al.  Garment capture from a photograph , 2015, Comput. Animat. Virtual Worlds.

[35]  Alexander M. Bronstein,et al.  Calculus of Nonrigid Surfaces for Geometry and Texture Manipulation , 2007, IEEE Transactions on Visualization and Computer Graphics.

[36]  Wolfgang Straßer,et al.  Continuum‐based Strain Limiting , 2009, Comput. Graph. Forum.

[37]  Kun Zhou,et al.  Single-view hair modeling for portrait manipulation , 2012, ACM Trans. Graph..

[38]  Tamara L. Berg,et al.  Paper Doll Parsing: Retrieving Similar Styles to Parse Clothing Items , 2013, 2013 IEEE International Conference on Computer Vision.

[39]  Derek Bradley,et al.  Wrinkling Captured Garments Using Space‐Time Data‐Driven Deformation , 2009, Comput. Graph. Forum.

[40]  Richard L. Grimsdale,et al.  Computer graphics techniques for modeling cloth , 1996, IEEE Computer Graphics and Applications.

[41]  Lin Wang,et al.  Garment modeling with a depth camera , 2015, ACM Trans. Graph..

[42]  Ronald Fedkiw,et al.  Robust treatment of collisions, contact and friction for cloth animation , 2002, SIGGRAPH Courses.

[43]  Andries Petrus Engelbrecht,et al.  A study of particle swarm optimization particle trajectories , 2006, Inf. Sci..

[44]  Jovan Popovic,et al.  Deformation transfer for triangle meshes , 2004, ACM Trans. Graph..

[45]  Camille Couprie,et al.  Learning Hierarchical Features for Scene Labeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Huamin Wang,et al.  Data-driven elastic models for cloth: modeling and measurement , 2011, ACM Trans. Graph..

[47]  Pieter Peers,et al.  Temporally coherent completion of dynamic shapes , 2012, TOGS.

[48]  Huamin Wang,et al.  Multi-resolution isotropic strain limiting , 2010, ACM Trans. Graph..

[49]  Ronan Collobert,et al.  Recurrent Convolutional Neural Networks for Scene Labeling , 2014, ICML.

[50]  Camillo J. Taylor,et al.  Reconstruction of Articulated Objects from Point Correspondences in a Single Uncalibrated Image , 2000, Comput. Vis. Image Underst..

[51]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[52]  Jerry Weil,et al.  The synthesis of cloth objects , 1986, SIGGRAPH.

[53]  Hans-Peter Seidel,et al.  Physically Based Tracking of Cloth , 2006 .

[54]  Ronald Fedkiw,et al.  Simulation of clothing with folds and wrinkles , 2003, SCA '03.

[55]  Romain Casati,et al.  Inverse Elastic Cloth Design with Contact and Friction , 2019 .

[56]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Dinesh Manocha,et al.  Fast collision detection for deformable models using representative-triangles , 2008, I3D '08.

[58]  Charlie C. L. Wang,et al.  Flexible shape control for automatic resizing of apparel products , 2012, Comput. Aided Des..

[59]  Ruigang Yang,et al.  Semantic Parametric Reshaping of Human Body Models , 2014, 2014 2nd International Conference on 3D Vision.

[60]  Gabriel Taubin,et al.  A signal processing approach to fair surface design , 1995, SIGGRAPH.

[61]  Alla Sheffer,et al.  Virtual Garments: A Fully Geometric Approach for Clothing Design , 2006, Comput. Graph. Forum.

[62]  Robert Bridson,et al.  Animating developable surfaces using nonconforming elements , 2008, ACM Trans. Graph..

[63]  Romain Casati,et al.  Inverse elastic shell design with contact and friction , 2018, ACM Trans. Graph..

[64]  Eitan Grinspun,et al.  To appear in the ACM SIGGRAPH conference proceedings Efficient Simulation of Inextensible Cloth , 2007 .

[65]  Yu Wang,et al.  Design automation for customized apparel products , 2005, Comput. Aided Des..

[66]  Hans-Peter Seidel,et al.  A Statistical Model of Human Pose and Body Shape , 2009, Comput. Graph. Forum.

[67]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[68]  Ankur Agarwal,et al.  Recovering 3D human pose from monocular images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[69]  Nadia Magnenat-Thalmann,et al.  A body and Garment Creation Method for an Internet Based Virtual Fitting Room. , 2002 .

[70]  Jernej Barbic,et al.  Skin microstructure deformation with displacement map convolution , 2015, ACM Trans. Graph..

[71]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[72]  Michael J. Black,et al.  Detailed Human Shape and Pose from Images , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[73]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.