Understanding and Recreating Visual Appearance Under Natural Illumination

The appearance of an outdoor scene is determined to a great extent by the prevailing illumination conditions. However, most practical computer vision applications treat illumination more as a nuisance rather than a source of signal. In this dissertation, we suggest that we should instead embrace illumination, even in the challenging, uncontrolled world of consumer photographs. Our first main contribution is an understanding of natural illumination from images. This is, in general, a hard problem given the wide appearance variation in scenes. Fortunately, natural illumination, while complex, is far from being completely arbitrary. It has a structure that is well understood in atmospheric optics, but which has hardly been exploited in vision and graphics. We introduce methods for automatically estimating the illumination conditions from two types of uncontrolled outdoor image datasets: webcams and single images. The variation in sun position and sky appearance over time can be exploited to obtain viewing and illumination geometry in webcam sequences. For single images, the sky is combined in a probabilistic way with other scene features such as cast shadows and shading on vertical surfaces and convex objects, as well as with illumination priors from large image collections. Our second main contribution is to exploit the knowledge of illumination in order to synthesize novel, realistic visual content. Instead of creating appearance using the traditional computer graphics pipeline, we propose to borrow the appearance of the world that is contained in existing photo collections and webcam datasets. We also demonstrate realistic all object insertion by creating plausible high-dynamic range environment maps. This can be done in image sequences, and even in single images, completely automatically. Addressing such questions has implications in a broad range of applications including intelligent transportation, surveillance, human-robot interaction, and digital entertainment.

[1]  J. Robson,et al.  Application of fourier analysis to the visibility of gratings , 1968, The Journal of physiology.

[2]  Wojciech Matusik,et al.  Factored time-lapse video , 2007, SIGGRAPH 2007.

[3]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[4]  Graham D. Finlayson,et al.  Simple Shadow Remova , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[5]  Ira Kemelmacher-Shlizerman,et al.  Photometric Stereo with General, Unknown Lighting , 2006, International Journal of Computer Vision.

[6]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[7]  Shree K. Nayar,et al.  Modeling the space of camera response functions , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Paul E. Debevec,et al.  Digitizing the Parthenon: Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination , 2004, VMV.

[9]  Steven J. Gortler,et al.  A perception-based color space for illumination-invariant image processing , 2008, SIGGRAPH 2008.

[10]  Alexei A. Efros,et al.  Detecting Ground Shadows in Outdoor Consumer Photographs , 2010, ECCV.

[11]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Hassan Foroosh,et al.  Estimating Geo-temporal Location of Stationary Cameras Using Shadow Trajectories , 2008, ECCV.

[13]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[14]  William T. Freeman,et al.  Learning Low-Level Vision , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[15]  Alexei A. Efros,et al.  Estimating natural illumination from a single outdoor image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[16]  Ian D. Reid,et al.  Single View Metrology , 2000, International Journal of Computer Vision.

[17]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[18]  Roberto Cipolla,et al.  Semantic Photo Synthesis , 2006, Comput. Graph. Forum.

[19]  Pietro Perona,et al.  Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[20]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[21]  Robert Pless,et al.  Geolocating Static Cameras , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[22]  Jian Sun,et al.  Drag-and-drop pasting , 2006, SIGGRAPH 2006.

[23]  David Salesin,et al.  Shadow matting and compositing , 2003, ACM Trans. Graph..

[24]  Dale Purves,et al.  The empirical basis of color perception , 2002, Consciousness and Cognition.

[25]  Shimon Ullman,et al.  Face Recognition: The Problem of Compensating for Changes in Illumination Direction , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[27]  Andrew Jones,et al.  Direct HDR capture of the sun and sky , 2004, AFRIGRAPH '04.

[28]  Katsushi Ikeuchi,et al.  Consistent surface color for texturing large objects in outdoor scenes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[29]  Alexei A. Efros,et al.  Using Multiple Segmentations to Discover Objects and their Extent in Image Collections , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[30]  Cheng Lu,et al.  On the removal of shadows from images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Katsushi Ikeuchi,et al.  Illumination normalization with time-dependent intrinsic images for video surveillance , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Cheng Lu,et al.  Intrinsic Images by Entropy Minimization , 2004, ECCV.

[33]  Wojciech Matusik,et al.  Image restoration using online photo collections , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[34]  David J. Kriegman,et al.  Nine points of light: acquiring subspaces for face recognition under variable lighting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[35]  Yoram Singer,et al.  Logistic Regression, AdaBoost and Bregman Distances , 2000, Machine Learning.

[36]  Hany Farid,et al.  Exposing Digital Forgeries in Complex Lighting Environments , 2007, IEEE Transactions on Information Forensics and Security.

[37]  Dani Lischinski,et al.  The Shadow Meets the Mask: Pyramid‐Based Shadow Removal , 2008, Comput. Graph. Forum.

[38]  Frédo Durand,et al.  Light mixture estimation for spatially varying white balance , 2008, SIGGRAPH 2008.

[39]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[40]  John Mardaljevic,et al.  Dynamic Daylight Performance Metrics for Sustainable Building Design , 2006 .

[41]  D. B. Judd,et al.  Spectral Distribution of Typical Daylight as a Function of Correlated Color Temperature , 1964 .

[42]  Masayuki Nakajima,et al.  Example-Based Color Stylization of Images , 2005, TAP.

[43]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[44]  Robert Pless,et al.  Toward Fully Automatic Geo-Location and Geo-Orientation of Static Outdoor Cameras , 2008, 2008 IEEE Workshop on Applications of Computer Vision.

[45]  Edward H. Adelson,et al.  Recovering reflectance and illumination in a world of painted polyhedra , 1993, 1993 (4th) International Conference on Computer Vision.

[46]  Jan-Michael Frahm,et al.  Radiometric calibration with illumination change for outdoor scene analysis , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Tom Duff,et al.  Compositing digital images , 1984, SIGGRAPH.

[49]  Ken-ichi Anjyo,et al.  Tour into the picture: using a spidery mesh interface to make animation from a single image , 1997, SIGGRAPH.

[50]  Alexei A. Efros,et al.  Recovering Surface Layout from an Image , 2007, International Journal of Computer Vision.

[51]  Jiandong Tian,et al.  Tricolor Attenuation Model for Shadow Detection , 2009, IEEE Transactions on Image Processing.

[52]  Alexei A. Efros,et al.  Webcam clip art: appearance and illuminant transfer from time-lapse sequences , 2009, SIGGRAPH 2009.

[53]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH '08.

[54]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[55]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Bruce A. Draper,et al.  Color Models for Outdoor Machine Vision , 2002, Comput. Vis. Image Underst..

[57]  Derek Hoiem,et al.  Seeing the world behind the image: Spatial layout for 3D scene understanding , 2007 .

[58]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[59]  Fabio Gagliardi Cozman,et al.  Robot localization using a computer vision sextant , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[60]  O. Sorkine,et al.  Color harmonization , 2006, SIGGRAPH 2006.

[61]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[62]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[63]  Michael Goesele,et al.  Multi-View Stereo for Community Photo Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[64]  Alexei A. Efros,et al.  Putting Objects in Perspective , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[65]  Silvio Savarese,et al.  3D generic object categorization, localization and pose estimation , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[66]  Erik Reinhard,et al.  Real-time color blending of rendered and captured video , 2004 .

[67]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[68]  Jian Sun,et al.  SkyFinder: attribute-based sky image search , 2009, SIGGRAPH 2009.

[69]  Brett Kennedy,et al.  Design and analysis of a sun sensor for planetary rover absolute heading detection , 2001, IEEE Trans. Robotics Autom..

[70]  Mingxuan Sun,et al.  Color matching and illumination estimation for urban scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[71]  Antonio Criminisi,et al.  TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-class Object Recognition and Segmentation , 2006, ECCV.

[72]  Daniel Cremers,et al.  An Integral Solution to Surface Evolution PDEs Via Geo-cuts , 2006, ECCV.

[73]  A. T. Young,et al.  Revised optical air mass tables and approximation formula. , 1989, Applied optics.

[74]  Heung-Yeung Shum,et al.  Radiometric calibration from a single image , 2004, CVPR 2004.

[75]  H H Bülthoff,et al.  A Prior for Global Convexity in Local Shape-from-Shading , 2001, Perception.

[76]  Graham D. Finlayson,et al.  Color by Correlation: A Simple, Unifying Framework for Color Constancy , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[77]  Frédo Durand,et al.  Noise-optimal capture for high dynamic range photography , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[78]  Ramesh C. Jain,et al.  Content Based Image Synthesis , 2004, CIVR.

[79]  Charless C. Fowlkes,et al.  Multiresolution Models for Object Detection , 2010, ECCV.

[80]  Shree K. Nayar,et al.  Priors for Large Photo Collections and What They Reveal about Cameras , 2008, ECCV.

[81]  Alexei A. Efros,et al.  Photo clip art , 2007, SIGGRAPH 2007.

[82]  Pascal Mamassian,et al.  Illusory motion from shadows , 1996, Nature.

[83]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[84]  James H Elder,et al.  Rapid Processing of Cast and Attached Shadows , 2004, Perception.

[85]  Wei Zhang,et al.  Video Compass , 2002, ECCV.

[86]  Gregory J. Ward,et al.  The RADIANCE lighting simulation and rendering system , 1994, SIGGRAPH.

[87]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2008 .

[88]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[89]  David W. Jacobs,et al.  In search of illumination invariants , 2001, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[90]  Jiejie Zhu,et al.  Learning to recognize shadows in monochromatic natural images , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[91]  Alexei A. Efros,et al.  Geometric context from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[92]  Taeone Kim,et al.  A practical single image based approach for estimating illumination distribution from shadows , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[93]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[94]  J. Koenderink,et al.  Light Direction from Shad(ow)ed Random Gaussian Surfaces , 2004, Perception.

[95]  Pat Hanrahan,et al.  A signal-processing framework for inverse rendering , 2001, SIGGRAPH.

[96]  Peter Shirley,et al.  A practical analytic model for daylight , 1999, SIGGRAPH.

[97]  Sylvain Paris,et al.  User-assisted intrinsic images , 2009, ACM Trans. Graph..

[98]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[99]  Tai-Pang Wu,et al.  A Bayesian approach for shadow extraction from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[100]  Steven M. Seitz,et al.  Finding paths through the world's photos , 2008, SIGGRAPH 2008.

[101]  Holly E. Rushmeier,et al.  Implementation and Analysis of an Image-Based Global Illumination Framework for Animated Environments , 1996, IEEE Trans. Vis. Comput. Graph..

[102]  P. Cavanagh.  The artist as neuroscientist , 2005, Nature.

[103]  Alexei A. Efros,et al.  IM2GPS: estimating geographic information from a single image , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[104]  Yee Whye Teh,et al.  Names and faces in the news , 2004, CVPR 2004.

[105]  Takeo Kanade,et al.  Bayesian color constancy for outdoor object recognition , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[106]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[107]  Robert Pless,et al.  Using cloud shadows to infer scene structure and camera calibration , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[108]  Mark S. Drew,et al.  Detecting Illumination in Images , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[109]  David J. Kriegman,et al.  Color Subspaces as Photometric Invariants , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[110]  I. Reda,et al.  Solar position algorithm for solar radiation applications , 2004 .

[111]  David A. McAllester,et al.  A discriminatively trained, multiscale, deformable part model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[112]  Thomas B. Moeslund,et al.  Detection and removal of chromatic moving shadows in surveillance scenarios , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[113]  Yan Ke,et al.  The Design of High-Level Features for Photo Quality Assessment , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[114]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[115]  Vladimir Kolmogorov,et al.  What metrics can be approximated by geo-cuts, or global optimization of length/area and flux , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[116]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[117]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[118]  Jitendra Malik,et al.  Recovering photometric properties of architectural scenes from photographs , 1998, SIGGRAPH.

[119]  Manuel Blum,et al.  Peekaboom: a game for locating objects in images , 2006, CHI.

[120]  Edward H. Adelson,et al.  Recovering intrinsic images from a single image , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[121]  Cheng Lu,et al.  Entropy Minimization for Shadow Removal , 2009, International Journal of Computer Vision.