Estimating the Natural Illumination Conditions from a Single Outdoor Image

Given a single outdoor image, we present a method for estimating the likely illumination conditions of the scene. In particular, we compute the probability distribution over the sun position and visibility. The method relies on a combination of weak cues that can be extracted from different portions of the image: the sky, the vertical surfaces, the ground, and the convex objects in the image. While no single cue can reliably estimate illumination by itself, each one can reinforce the others to yield a more robust estimate. This is combined with a data-driven prior computed over a dataset of 6 million photos. We present quantitative results on a webcam dataset with annotated sun positions, as well as quantitative and qualitative results on consumer-grade photographs downloaded from Internet. Based on the estimated illumination, we show how to realistically insert synthetic 3-D objects into the scene, and how to transfer appearance across images while keeping the illumination consistent.

[1]  Wojciech Matusik,et al.  What do color changes reveal about an outdoor scene? , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[3]  Tai-Pang Wu,et al.  A Bayesian approach for shadow extraction from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[4]  Takeo Kanade,et al.  Bayesian color constancy for outdoor object recognition , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[5]  Alexei A. Efros,et al.  Webcam clip art: appearance and illuminant transfer from time-lapse sequences , 2009, ACM Trans. Graph..

[6]  Shree K. Nayar,et al.  A class of photometric invariants: separating material from shape and illumination , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[7]  Taeone Kim,et al.  A practical single image based approach for estimating illumination distribution from shadows , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[8]  Alexei A. Efros,et al.  What Do the Sun and the Sky Tell Us About the Camera? , 2010, International Journal of Computer Vision.

[9]  Wojciech Matusik,et al.  Image restoration using online photo collections , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[10]  Ronen Basri,et al.  Photometric stereo with general, unknown lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[11]  Jiejie Zhu,et al.  Learning to recognize shadows in monochromatic natural images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Robert Pless,et al.  Consistent Temporal Variations in Many Outdoor Scenes , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  I. Reda,et al.  Solar position algorithm for solar radiation applications , 2004 .

[14]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[15]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[16]  H H Bülthoff,et al.  A Prior for Global Convexity in Local Shape-from-Shading , 2001, Perception.

[17]  D. Mills.  Advances in solar thermal electricity technology , 2004 .

[18]  Alexei A. Efros,et al.  Automatic photo pop-up , 2005, SIGGRAPH 2005.

[19]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Katsushi Ikeuchi,et al.  Illumination from Shadows , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Bruce A. Draper,et al.  Color Models for Outdoor Machine Vision , 2002, Comput. Vis. Image Underst..

[22]  Steven J. Gortler,et al.  A perception-based color space for illumination-invariant image processing , 2008, ACM Trans. Graph..

[23]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[24]  A. T. Young,et al.  Revised optical air mass tables and approximation formula. , 1989, Applied optics.

[25]  Ron O Dror,et al.  Statistical characterization of real-world illumination. , 2004, Journal of vision.

[26]  Alexei A. Efros,et al.  Recovering Occlusion Boundaries from a Single Image , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[27]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[28]  Mark S. Drew,et al.  Diagonal transforms suffice for color constancy , 1993, 1993 (4th) International Conference on Computer Vision.

[29]  Cheng Lu,et al.  Intrinsic Images by Entropy Minimization , 2004, ECCV.

[30]  Gregory J. Ward,et al.  The RADIANCE lighting simulation and rendering system , 1994, SIGGRAPH.

[31]  Roberto Manduchi,et al.  Learning Outdoor Color Classification , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  R. M. Laws,et al.  Elephant seals : population ecology, behavior, and physiology , 1994 .

[33]  Alexei A. Efros,et al.  Photo clip art , 2007, ACM Trans. Graph..

[34]  Todd E. Zickler,et al.  Blind Reflectometry , 2010, ECCV.

[35]  Bruce A. Maxwell,et al.  A bi-illuminant dichromatic reflection model for understanding images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[37]  Pietro Perona,et al.  Pedestrian detection: A benchmark , 2009, CVPR.

[38]  Mark S. Drew,et al.  Detecting Illumination in Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[39]  Glenn Healey,et al.  Analyzing the spectral dimensionality of outdoor visible and near-infrared illumination functions , 1998 .

[40]  Paul E. Debevec,et al.  Digitizing the Parthenon: Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination , 2004, VMV.

[41]  Peter Shirley,et al.  A practical analytic model for daylight , 1999, SIGGRAPH.

[42]  David J. Kriegman,et al.  Color Subspaces as Photometric Invariants , 2006, CVPR.

[43]  D. B. Judd,et al.  Spectral Distribution of Typical Daylight as a Function of Correlated Color Temperature , 1964 .

[44]  Reza Rajimehr,et al.  Subliminal attentional modulation in crowding condition , 2004 .

[45]  Mingxuan Sun,et al.  Color matching and illumination estimation for urban scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[46]  Jiandong Tian,et al.  Tricolor Attenuation Model for Shadow Detection , 2009, IEEE Transactions on Image Processing.

[47]  Mark S. Drew,et al.  Removing Shadows from Images , 2002, ECCV.

[48]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[49]  Wei Zhang,et al.  Video Compass , 2002, ECCV.

[50]  J. Koenderink,et al.  Light Direction from Shad(ow)ed Random Gaussian Surfaces , 2004, Perception.

[51]  R. Bird.  A simple, solar spectral model for direct-normal and diffuse horizontal irradiance , 1984 .

[52]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[53]  Alexei A. Efros,et al.  IM2GPS: estimating geographic information from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[54]  Hassan Foroosh,et al.  Estimating Geo-temporal Location of Stationary Cameras Using Shadow Trajectories , 2008, ECCV.

[55]  Charless C. Fowlkes,et al.  Multiresolution Models for Object Detection , 2010, ECCV.

[56]  John Platt,et al.  Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods , 1999 .

[57]  Alexei A. Efros,et al.  Estimating natural illumination from a single outdoor image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[58]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[59]  Paul Debevec.  Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[60]  James F. Blinn,et al.  Texture and reflection in computer generated images , 1998 .

[61]  J. Michalsky,et al.  All-weather model for sky luminance distribution—Preliminary configuration and validation , 1993 .

[62]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[63]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[64]  Stephen Lin,et al.  Multiple-cue illumination estimation in textured scenes , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[65]  P. Cavanagh.  The artist as neuroscientist , 2005, Nature.

[66]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[67]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[68]  John Mardaljevic,et al.  Dynamic Daylight Performance Metrics for Sustainable Building Design , 2006 .

[69]  Yoram Singer,et al.  Logistic Regression, AdaBoost and Bregman Distances , 2000, Machine Learning.

[70]  David W. Jacobs,et al.  In search of illumination invariants , 2001, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[71]  K. Ikeuchi,et al.  Reflectance analysis under solar illumination , 1994, Proceedings of the Workshop on Physics-Based Modeling in Computer Vision.

[72]  Glenn Healey,et al.  Models and methods for automated material identification in hyperspectral imagery acquired under unknown illumination and atmospheric conditions , 1999, IEEE Trans. Geosci. Remote. Sens..

[73]  Alexei A. Efros,et al.  Detecting Ground Shadows in Outdoor Consumer Photographs , 2010, ECCV.