Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View

Recent years have seen the development of mature solutions for reconstructing deformable surfaces from a single image, provided that they are relatively well-textured. By contrast, recovering the 3D shape of texture-less surfaces remains an open problem, and essentially relates to Shape-from-Shading. In this paper, we introduce a data-driven approach to this problem. We introduce a general framework that can predict diverse 3D representations, such as meshes, normals, and depth maps. Our experiments show that meshes are ill-suited to handle texture-less 3D reconstruction in our context. Furthermore, we demonstrate that our approach generalizes well to unseen objects, and that it yields higher-quality reconstructions than a state-of-the-art SfS technique, particularly in terms of normal estimates. Our reconstructions accurately model the fine details of the surfaces, such as the creases of a T-Shirt worn by a person.

[1]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Iasonas Kokkinos,et al.  Face Normals "In-the-Wild" Using Fully Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Bill Freeman,et al.  Shape and Illumination from Shading using the Generic Viewpoint Assumption , 2014, NIPS.

[4]  Pascal Fua,et al.  Monocular 3D Reconstruction of Locally Textured Surfaces , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[7]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Namil Kim,et al.  Fine-Scale Surface Normal Estimation Using a Single NIR Image , 2016, ECCV.

[9]  Antoni B. Chan,et al.  Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[10]  Yaser Sheikh,et al.  Modeling Facial Geometry Using Compositional VAEs , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[11]  Stefan Roth,et al.  Discriminative shape from shading in uncalibrated illumination , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[13]  M. B. Stegmann,et al.  A Brief Introduction to Statistical Shape Analysis , 2002 .

[14]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  Ersin Yumer,et al.  Neural Face Editing with Intrinsic Image Disentangling , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Pascal Fua,et al.  Template-Based Monocular 3D Shape Recovery Using Laplacian Meshes , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Olivier D. Faugeras,et al.  Shape From Shading , 2006, Handbook of Mathematical Models in Computer Vision.

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[21]  Allan D. Jepson,et al.  Polynomial shape from shading , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Jiajun Wu,et al.  Self-Supervised Intrinsic Image Decomposition , 2017, NIPS.

[23]  Nazar Khan,et al.  Training many-parameter shape-from-shading models using a surface database , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[24]  Xiaoming Liu,et al.  Large-Pose Face Alignment via CNN-Based Dense 3D Model Fitting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Jian Shi,et al.  Learning Non-Lambertian Object Intrinsics Across ShapeNet Categories , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Carlos D. Castillo,et al.  SfSNet: Learning Shape, Reflectance and Illuminance of Faces 'in the Wild' , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Vincent Lepetit,et al.  Geometry-Aware Network for Non-rigid Shape Prediction from a Single View , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[29]  Fei Wang,et al.  Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces , 2016, ECCV.

[30]  Matan Sela,et al.  Learning Detailed Face Reconstruction from a Single Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Markus H. Gross,et al.  DeepGarment : 3D Garment Shape Estimation from a Single Image , 2017, Comput. Graph. Forum.

[32]  Joshua B. Tenenbaum,et al.  Deep Convolutional Inverse Graphics Network , 2015, NIPS.

[33]  Jean-Denis Durou,et al.  Numerical methods for shape-from-shading: A new survey with benchmarks , 2008, Comput. Vis. Image Underst..

[34]  Ruigang Yang,et al.  Examplar-based Shape from Shading , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[35]  Pascal Fua,et al.  Variable albedo surface reconstruction from stereo and shape from shading , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[36]  Olivier D. Faugeras,et al.  Shape from shading: a well-posed problem? , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[37]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Jitendra Malik,et al.  High-frequency shape and albedo from shading using natural image statistics , 2011, CVPR 2011.

[39]  Ronen Basri,et al.  From Shading to Local Shape , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Ronen Basri,et al.  Example Based 3D Reconstruction from Single 2D Images , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[41]  In-So Kweon,et al.  Simultaneous Estimation of Near IR BRDF and Fine-Scale Surface Geometry , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.