3D Virtual Garment Modeling from RGB Images

We present a novel and efficient approach that automatically constructs 3D models for garments using only RGB images. Most of the previous methods deal with input photos in which the garment is on a human body or mannequin. Our approach can work with various types of input garment photos: photos in which the garment is worn by a model, or photos in which the garment is laid on a flat surface. To construct a complete 3D model, our approach requires minimum two images as input: one front view and one rear view. We propose a multi-task learning network called JFNet that jointly identifies the landmarks of the garment as well as parses the garment into semantic part segments. The predicted landmarks are used to estimate the garment size thus a template mesh can be deformed accordingly to construct the 3D mesh model. Color and textures of the model are extracted by exploiting the parsed semantic parts from input images. Our approach can be applied in various Virtual Reality and Mixed Reality applications involving garment modeling.

[1]  Hyeong-Seok Ko,et al.  Garment capture from a photograph , 2015, Comput. Animat. Virtual Worlds.

[2]  Xiaogang Wang,et al.  Unconstrained Fashion Landmark Detection via Hierarchical Recurrent Transformer Networks , 2017, ACM Multimedia.

[3]  Marcus A. Magnor,et al.  Video Based Reconstruction of 3D People Models , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[4]  Hao Li,et al.  SiCloPe: Silhouette-Based Clothed People , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Varun Ramakrishna,et al.  Pose Machines: Articulated Pose Estimation via Inference Machines , 2014, ECCV.

[6]  Larry S. Davis,et al.  VITON: An Image-Based Virtual Try-on Network , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Liang Lin,et al.  Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Marcus A. Magnor,et al.  Learning to Reconstruct People in Clothing From a Single RGB Camera , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Xiaogang Wang,et al.  Pyramid Scene Parsing Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Michael J. Black,et al.  ClothCap , 2017, ACM Trans. Graph..

[11]  Niloy J. Mitra,et al.  Learning a shared shape space for multimodal garment design , 2018, ACM Trans. Graph..

[12]  Yuandong Tian,et al.  Exploring the Spatial Hierarchy of Mixture Models for Human Pose Estimation , 2012, ECCV.

[13]  Peter V. Gehler,et al.  A Generative Model of People in Clothing , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Yu Wang,et al.  Design automation for customized apparel products , 2005, Comput. Aided Des..

[15]  Jawadul H. Bappy,et al.  Pose Guided Fashion Image Synthesis Using Deep Generative Model , 2019, ArXiv.

[16]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Ke Wang,et al.  Physics-Inspired Garment Recovery from a Single-View Image , 2018, ACM Trans. Graph..

[19]  Eitan Grinspun,et al.  Parsing sewing patterns into 3D garments , 2013, ACM Trans. Graph..

[20]  David A. Forsyth,et al.  Capturing and animating occluded cloth , 2007, ACM Trans. Graph..

[21]  Thomas W. Sederberg,et al.  Free-form deformation of solid geometric models , 1986, SIGGRAPH.

[22]  Alla Sheffer,et al.  Virtual Garments: A Fully Geometric Approach for Clothing Design , 2006, Comput. Graph. Forum.

[23]  Charlie C. L. Wang,et al.  Flexible shape control for automatic resizing of apparel products , 2012, Comput. Aided Des..

[24]  Qionghai Dai,et al.  SimulCap : Single-View Human Performance Capture With Cloth Simulation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Derek Bradley,et al.  Markerless garment capture , 2008, SIGGRAPH 2008.

[26]  Qionghai Dai,et al.  DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Iasonas Kokkinos,et al.  UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Alla Sheffer,et al.  Context-aware garment modeling from sketches , 2011, Comput. Graph..

[29]  Marcus A. Magnor,et al.  Detailed Human Avatars from Monocular Video , 2018, 2018 International Conference on 3D Vision (3DV).

[30]  Bernt Schiele,et al.  Pictorial structures revisited: People detection and articulated pose estimation , 2009, CVPR.

[31]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[32]  Scott Schaefer,et al.  Image deformation using moving least squares , 2006, ACM Trans. Graph..

[33]  Rama Chellappa,et al.  HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Ming Yang,et al.  Instance-level Human Parsing via Part Grouping Network , 2018, ECCV.

[35]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Michael J. Black,et al.  Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Bin Zhou,et al.  Garment Modeling from a Single Image , 2013, Comput. Graph. Forum.

[38]  Christian Theobalt,et al.  LiveCap , 2018, ACM Trans. Graph..

[39]  Iasonas Kokkinos,et al.  Dense Pose Transfer , 2018, ECCV.

[40]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Lin Wang,et al.  Garment modeling with a depth camera , 2015, ACM Trans. Graph..

[42]  Xiaogang Wang,et al.  Fashion Landmark Detection in the Wild , 2016, ECCV.

[43]  Markus H. Gross,et al.  DeepGarment : 3D Garment Shape Estimation from a Single Image , 2017, Comput. Graph. Forum.

[44]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.