Radiometric Scene Decomposition: Scene Reflectance, Illumination, and Geometry from RGB-D Images

Recovering the radiometric properties of a scene (i.e., the reflectance, illumination, and geometry) is a long-sought ability of computer vision that can provide invaluable information for a wide range of applications. Deciphering the radiometric ingredients from the appearance of a real-world scene, as opposed to a single isolated object, is particularly challenging as it generally consists of various objects with different material compositions exhibiting complex reflectance and light interactions that are also part of the illumination. We introduce the first method for radiometric decomposition of real-world scenes that handles those intricacies. We use RGB-D images to bootstrap geometry recovery and simultaneously recover the complex reflectance and natural illumination while refining the noisy initial geometry and segmenting the scene into different material regions. Most important, we handle real-world scenes consisting of multiple objects of unknown materials, which necessitates the modeling of spatially-varying complex reflectance, natural illumination, texture, interreflection and shadows. We systematically evaluate the effectiveness of our method on synthetic scenes and demonstrate its application to real-world scenes. The results show that rich radiometric information can be recovered from RGB-D images and demonstrate a new role RGB-D sensors can play for general scene understanding tasks.

[1]  Oliver Wang,et al.  Material classification using BRDF slices , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Ko Nishino Directional statistics BRDF model , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[3]  Alfred M. Bruckstein,et al.  RGBD-fusion: Real-time high precision depth recovery , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Chao Liu,et al.  Discriminative illumination: Per-pixel classification of raw materials based on optimal projections of spectral BRDF , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Ko Nishino,et al.  Multiview Shape and Reflectance from Natural Illumination , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Pieter Peers,et al.  Appearance-from-motion , 2014, ACM Trans. Graph..

[7]  Ko Nishino,et al.  Single image multimaterial estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Jitendra Malik,et al.  Intrinsic Scene Properties from a Single RGB-D Image , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Stephen Lin,et al.  Shading-Based Shape Refinement of RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Kun Zhou,et al.  Simultaneous Localization and Appearance Estimation with a Consumer RGB-D Camera , 2016, IEEE Transactions on Visualization and Computer Graphics.

[11]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Ko Nishino,et al.  Directional statistics-based reflectance model for isotropic bidirectional reflectance distribution functions. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[13]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[14]  Kalyan Sunkavalli,et al.  Automatic Scene Inference for 3D Object Compositing , 2014, ACM Trans. Graph..

[15]  Andrew W. Fitzgibbon,et al.  KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera , 2011, UIST.

[16]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[17]  James T. Kajiya,et al.  The rendering equation , 1986, SIGGRAPH.

[18]  Ko Nishino,et al.  Shape and Reflectance Estimation in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Ko Nishino,et al.  Reflectance and Natural Illumination from a Single Image , 2012, ECCV.

[20]  Greg Humphreys,et al.  Physically Based Rendering, Second Edition: From Theory To Implementation , 2010 .

[21]  Steven M. Seitz,et al.  Shape and spatially-varying BRDFs from photometric stereo , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.