Exploring Material Recognition for Estimating Reflectance and Illumination From a Single Image

In this paper, we propose a novel approach for recovering illumination and reflectance from a single image. Our approach relies on the assumption that the surface geometry has already been reconstructed and a-priori knowledge in form of a database of digital material models is available. The first step of our technique consists in recognizing the respective material in the image using synthesized training data based on the given material database. Subsequently, the illumination conditions are estimated based on the recognized material and the surface geometry. Using this novel strategy we demonstrate that reflectance and illumination can be estimated reliably for several materials that are beyond simple Lambertian surface reflectance behavior because of exhibiting mesoscopic effects such as interreflections and shadows.

[1]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[2]  Cordelia Schmid,et al.  Aggregating local descriptors into a compact image representation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  Christopher Schwartz,et al.  Design and Implementation of Practical Bidirectional Texture Function Measurement Devices Focusing on the Developments at the University of Bonn , 2014, Sensors.

[4]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[5]  Mark S. Drew,et al.  Recovering Shading from Color Images , 1992, ECCV.

[6]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[7]  Edward H. Adelson,et al.  Exploring features in a Bayesian framework for material recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Reinhard Klein,et al.  Advances in geometry and reflectance acquisition (course notes) , 2015, SIGGRAPH Asia Courses.

[9]  Adrien Bousseau,et al.  Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views , 2013, IEEE Trans. Vis. Comput. Graph..

[10]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Michael Goesele,et al.  Advances in Geometry and Reflectance Acquisition , 2016, Eurographics.

[12]  Christopher Schwartz,et al.  Integrated High-Quality Acquisition of Geometry and Appearance for Cultural Heritage , 2011, VAST.

[13]  Michal Haindl,et al.  Advanced textural representation of materials appearance , 2011, SA '11.

[14]  Tien-Tsin Wong,et al.  Image-based Rendering with Controllable Illumination , 1997, Rendering Techniques.

[15]  Michael Weinmann,et al.  Material Recognition for Efficient Acquisition of Geometry and Reflectance , 2014, ECCV Workshops.

[16]  Andrew Blake,et al.  Boundary conditions for lightness computation in Mondrian World , 1985, Comput. Vis. Graph. Image Process..

[17]  Bruno Lévy,et al.  ABF++: fast and robust angle based flattening , 2005, TOGS.

[18]  Xuelong Li,et al.  Intrinsic images using optimization , 2011, CVPR 2011.

[19]  Ko Nishino,et al.  Reflectance and Natural Illumination from a Single Image , 2012, ECCV.

[20]  Peter V. Gehler,et al.  Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance , 2011, NIPS.

[21]  Jean-François Lalonde,et al.  Lighting Estimation in Outdoor Image Collections , 2014, 2014 2nd International Conference on 3D Vision.

[22]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[23]  Florent Perronnin,et al.  Fisher Kernels on Visual Vocabularies for Image Categorization , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[25]  Ko Nishino,et al.  Shape and Reflectance from Natural Illumination , 2012, ECCV.

[26]  Yaser Sheikh,et al.  3D object manipulation in a single photograph using stock 3D models , 2014, ACM Trans. Graph..

[27]  Stephen Lin,et al.  Unified Depth Prediction and Intrinsic Image Decomposition from a Single Image via Joint Convolutional Neural Fields , 2016, ECCV.

[28]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[29]  H. Barrow,et al.  RECOVERING INTRINSIC SCENE CHARACTERISTICS FROM IMAGES , 1978 .

[30]  Roberto Scopigno,et al.  A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions , 2012, Comput. Graph. Forum.

[31]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Shree K. Nayar,et al.  Reflectance and Texture of Real-World Surfaces Authors , 1997, CVPR 1997.

[33]  Peter F. Sturm,et al.  Estimating Photometric Properties from Image Collections , 2013, Journal of Mathematical Imaging and Vision.

[34]  Michael Weinmann,et al.  Material Classification Based on Training Data Synthesized Using a BTF Database , 2014, ECCV.

[35]  Stephen Lin,et al.  Intrinsic image decomposition with non-local texture cues , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Kalyan Sunkavalli,et al.  Automatic Scene Inference for 3D Object Compositing , 2014, ACM Trans. Graph..