Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automatically-selected keyframes, and their camera poses along with material and scene lighting. To this end, we propose a joint surface reconstruction approach that is based on Shape-from-Shading (SfS) techniques and utilizes the estimation of spatially-varying spherical harmonics (SVSH) from subvolumes of the reconstructed scene. Through extensive examples and evaluations, we demonstrate that our method dramatically increases the level of detail in the reconstructed scene geometry and contributes highly to consistent surface texture recovery.

[1]  In-So Kweon,et al.  High Quality Shape from a Single RGB-D Image under Uncalibrated Natural Illumination , 2013, 2013 IEEE International Conference on Computer Vision.

[2]  Derek Bradley,et al.  Improved Reconstruction of Deforming Surfaces by Cancelling Ambient Occlusion , 2012, ECCV.

[3]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[4]  Patricia Ladret,et al.  The blur effect: perception and estimation with a new no-reference perceptual blur metric , 2007, Electronic Imaging.

[5]  MalikJitendra,et al.  Intrinsic Scene Properties from a Single RGB-D Image , 2016 .

[6]  Alfred M. Bruckstein,et al.  RGBD-fusion: Real-time high precision depth recovery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  G. Stiny.  Shape , 1999 .

[8]  Hans-Peter Seidel,et al.  Shading-based dynamic shape refinement from multi-view video under general illumination , 2011, 2011 International Conference on Computer Vision.

[9]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[10]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[11]  Kun Zhou,et al.  Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera , 2016, IEEE Transactions on Visualization and Computer Graphics.

[12]  Jitendra Malik,et al.  Intrinsic Scene Properties from a Single RGB-D Image , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Vladlen Koltun,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014, ACM Trans. Graph..

[15]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Matthias Nießner,et al.  Real-time 3D reconstruction at scale using voxel hashing , 2013, ACM Trans. Graph..

[17]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[18]  Daniel Cremers,et al.  Large-Scale Multi-resolution Surface Reconstruction from RGB-D Sequences , 2013, 2013 IEEE International Conference on Computer Vision.

[19]  Ko Nishino,et al.  Radiometric Scene Decomposition: Scene Reflectance, Illumination, and Geometry from RGB-D Images , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[20]  Seungyong Lee,et al.  Texture map generation for 3D reconstructed scenes , 2016, The Visual Computer.

[21]  Jörg Stückler,et al.  Super-resolution Keyframe Fusion for 3D Modeling with High-Quality Textures , 2015, 2015 International Conference on 3D Vision.

[22]  Michael F. Cohen,et al.  Emptying, refurnishing, and relighting indoor spaces , 2016, ACM Trans. Graph..

[23]  Shahram Izadi,et al.  Real-time shading-based refinement for consumer depth cameras , 2014, ACM Trans. Graph..

[24]  Matthias Nießner,et al.  Shading-based refinement on volumetric signed distance functions , 2015, ACM Trans. Graph..

[25]  Christian Theobalt,et al.  On-set performance capture of multiple actors with a stereo camera , 2013, ACM Trans. Graph..

[26]  Berthold K. P. Horn.  Obtaining shape from shading information , 1989 .

[27]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Daniel Cremers,et al.  A Super-Resolution Framework for High-Accuracy Multiview Reconstruction , 2013, International Journal of Computer Vision.

[29]  Marcus A. Magnor,et al.  Sampling based scene-space video processing , 2015, ACM Trans. Graph..

[30]  Matthias Nießner,et al.  BundleFusion , 2016, TOGS.