Learning Shape-from-Shading for Deformable Surfaces

Recent years have seen the development of mature solutions for reconstructing deformable surfaces from a single image, provided that they are relatively well-textured. By contrast, recovering the 3D shape of texture-less surfaces remains an open problem, and essentially relates to Shape-from-Shading. In this paper, we introduce a data-driven approach to this problem. We introduce a general framework that can predict diverse 3D representations, such as meshes, normals, and depth maps. Our experiments show that, in contrast with the well-textured scenario, meshes a ill-suited to handle texture-less 3D reconstruction. Furthermore, we demonstrate that our approach generalizes well to unseen objects, and that it yields higher-quality reconstructions than a state-of-the-art SfS technique, particularly in terms of normal estimates. Our reconstructions accurately model the fine details of the surfaces, such as the creases of a T-Shirt worn by a person.

[1]  Vincent Lepetit,et al.  Geometry-Aware Network for Non-rigid Shape Prediction from a Single View , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[3]  Mubarak Shah,et al.  Tracking Multiple Occluding People by Localizing on Multiple Scene Planes , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Ronen Basri,et al.  Example Based 3D Reconstruction from Single 2D Images , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[5]  In-So Kweon,et al.  Simultaneous Estimation of Near IR BRDF and Fine-Scale Surface Geometry , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Iasonas Kokkinos,et al.  Face Normals "In-the-Wild" Using Fully Convolutional Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Ronen Basri,et al.  From Shading to Local Shape , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Bill Freeman,et al.  Shape and Illumination from Shading using the Generic Viewpoint Assumption , 2014, NIPS.

[9]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Olivier D. Faugeras,et al.  Shape from shading: a well-posed problem? , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[11]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Jitendra Malik,et al.  High-frequency shape and albedo from shading using natural image statistics , 2011, CVPR 2011.

[13]  Pascal Fua,et al.  Monocular 3D Reconstruction of Locally Textured Surfaces , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  M. B. Stegmann,et al.  A Brief Introduction to Statistical Shape Analysis , 2002 .

[15]  Jean-Denis Durou,et al.  Numerical methods for shape-from-shading: A new survey with benchmarks , 2008, Comput. Vis. Image Underst..

[16]  Ersin Yumer,et al.  Neural Face Editing with Intrinsic Image Disentangling , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Pascal Fua,et al.  Template-Based Monocular 3D Shape Recovery Using Laplacian Meshes , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Ruigang Yang,et al.  Examplar-based Shape from Shading , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[19]  Namil Kim,et al.  Fine-Scale Surface Normal Estimation Using a Single NIR Image , 2016, ECCV.

[20]  Pascal Fua,et al.  Variable albedo surface reconstruction from stereo and shape from shading , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[21]  Pascal Fua,et al.  Linear Local Models for Monocular Reconstruction of Deformable Surfaces , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Fei Wang,et al.  Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces , 2016, ECCV.

[23]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[24]  Edward H. Adelson,et al.  Shape estimation in natural illumination , 2011, CVPR 2011.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Stefan Roth,et al.  Discriminative shape from shading in uncalibrated illumination , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[28]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Berthold K. P. Horn,et al.  Shape from shading , 1989 .

[30]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[31]  Allan D. Jepson,et al.  Polynomial shape from shading , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Joshua B. Tenenbaum,et al.  Deep Convolutional Inverse Graphics Network , 2015, NIPS.

[33]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[34]  Markus H. Gross,et al.  DeepGarment : 3D Garment Shape Estimation from a Single Image , 2017, Comput. Graph. Forum.

[35]  Adrien Bartoli,et al.  On template-based reconstruction from a single view: Analytical solutions and proofs of well-posedness for developable, isometric and conformal surfaces , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.