What is Around the Camera?

How much does a single image reveal about the environment it was taken in? In this paper, we investigate how much of that information can be retrieved from a foreground object, combined with the background (i.e. the visible part of the environment). Assuming it is not perfectly diffuse, the foreground object acts as a complexly shaped andfar-from-perfect mirror An additional challenge is that its appearance confounds the light coming from the environment with the unknown materials it is made of. We propose a learning-based approach to predict the environment from multiple reflectance maps that are computed from approximate surface normals. The proposed method allows us to jointly model the statistics of environments and material properties. We train our system from synthesized training data, but demonstrate its applicability to real-world data. Interestingly, our analysis shows that the information obtained from objects made out of multiple materials often is complementary and leads to better performance.

[1]  Katsushi Ikeuchi,et al.  Illumination from Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Paul E. Debevec,et al.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 1998, SIGGRAPH '08.

[3]  B K Horn,et al.  Calculating the reflectance map. , 1979, Applied optics.

[4]  G. Stiny.  Shape , 1999 .

[5]  Antonio Torralba,et al.  Accidental Pinhole and Pinspeck Cameras , 2014, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Stella X. Yu,et al.  Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Mario Fritz,et al.  Image-Based Synthesis and Re-synthesis of Viewpoints Guided by 3D Models , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Jitendra Malik,et al.  Intrinsic Scene Properties from a Single RGB-D Image , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Abhinav Gupta,et al.  Designing deep networks for surface normal estimation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Mario Fritz,et al.  Novel Views of Objects from a Single Image , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Trevor Darrell,et al.  Constrained Structured Regression with Convolutional Neural Networks , 2015, ArXiv.

[13]  Ravi Ramamoorthi,et al.  A Theory Of Frequency Domain Invariants: Spherical Harmonic Identities for BRDF/Lighting Transfer and Image Consistency , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Alexei A. Efros,et al.  IM2GPS: estimating geographic information from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[16]  Chunhua Shen,et al.  Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Shree K. Nayar,et al.  Corneal Imaging System: Environment from Eyes , 2006, International Journal of Computer Vision.

[18]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[19]  Stefan Roth,et al.  Discriminative shape from shading in uncalibrated illumination , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Philip Dutré,et al.  The influence of shape on the perception of material reflectance , 2007, ACM Trans. Graph..

[21]  Luc Van Gool,et al.  DeLight-Net: Decomposing Reflectance Maps into Specular Materials and Natural Illumination , 2016, ArXiv.

[22]  Peter-Pike J. Sloan,et al.  The Lit Sphere: A Model for Capturing NPR Shading from Art , 2001, Graphics Interface.

[23]  Ravi Ramamoorthi,et al.  Reflectance sharing: predicting appearance from a sparse set of images of a known shape , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[25]  Yinda Zhang,et al.  FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[27]  Adriana Kovashka,et al.  Seeing Behind the Camera: Identifying the Authorship of a Photograph , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Vladlen Koltun,et al.  A Simple Model for Intrinsic Image Decomposition with Depth Cues , 2013, 2013 IEEE International Conference on Computer Vision.

[29]  Hans-Peter Seidel,et al.  Image-based reconstruction of spatial appearance and geometric detail , 2003, TOGS.

[30]  Michael F. Cohen,et al.  Emptying, refurnishing, and relighting indoor spaces , 2016, ACM Trans. Graph..

[31]  Alexei A. Efros,et al.  Estimating the Natural Illumination Conditions from a Single Outdoor Image , 2012, International Journal of Computer Vision.

[32]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[34]  Ko Nishino,et al.  Reflectance and Natural Illumination from a Single Image , 2012, ECCV.

[35]  H. Barrow,et al.  RECOVERING INTRINSIC SCENE CHARACTERISTICS FROM IMAGES , 1978 .

[36]  Kalyan Sunkavalli,et al.  Automatic Scene Inference for 3D Object Compositing , 2014, ACM Trans. Graph..

[37]  Shree K. Nayar,et al.  Reflectance and texture of real-world surfaces , 1999, TOGS.

[38]  Yannick Hold-Geoffroy,et al.  Deep Outdoor Illumination Estimation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Ko Nishino,et al.  Radiometric Scene Decomposition: Scene Reflectance, Illumination, and Geometry from RGB-D Images , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[41]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[43]  Edward H. Adelson,et al.  Statistics of real-world illumination , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[44]  Ralph R. Martin,et al.  BiggerPicture: data-driven image extrapolation using graph matching , 2014, ACM Trans. Graph..

[45]  Luc Van Gool,et al.  A Gaussian Process Latent Variable Model for BRDF Inference , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[46]  Ko Nishino,et al.  Single image multimaterial estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[48]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.