Webcam clip art: appearance and illuminant transfer from time-lapse sequences

Webcams placed all over the world observe and record the visual appearance of a variety of outdoor scenes over long periods of time. The recorded time-lapse image sequences cover a wide range of illumination and weather conditions -- a vast untapped resource for creating visual realism. In this work, we propose to use a large repository of webcams as a "clip art" library from which users may transfer scene appearance (objects, scene backdrops, outdoor illumination) into their own time-lapse sequences or even single photographs. The goal is to combine the recent ideas from data-driven appearance transfer techniques with a general and theoretically-grounded physically-based illumination model. To accomplish this, the paper presents three main research contributions: 1) a new, high-quality outdoor webcam database that has been calibrated radiometrically and geometrically; 2) a novel approach for matching illuminations across different scenes based on the estimation of the properties of natural illuminants (sun, sky, weather and clouds), the camera geometry, and illumination-dependent scene features; 3) a new algorithm for generating physically plausible high dynamic range environment maps for each frame in a webcam sequence.

[1]  Andrew Gardner,et al.  A lighting reproduction approach to live-action compositing , 2002, SIGGRAPH.

[2]  Ken-ichi Anjyo,et al.  Tour into the picture: using a spidery mesh interface to make animation from a single image , 1997, SIGGRAPH.

[3]  Andrew Jones,et al.  Direct HDR capture of the sun and sky , 2004, AFRIGRAPH '04.

[4]  Shree K. Nayar,et al.  Face swapping: automatically replacing faces in photographs , 2008, SIGGRAPH 2008.

[5]  A. T. Young,et al.  Revised optical air mass tables and approximation formula. , 1989, Applied optics.

[6]  Jan-Michael Frahm,et al.  Radiometric calibration with illumination change for outdoor scene analysis , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[8]  Alexei A. Efros,et al.  What Do the Sun and the Sky Tell Us About the Camera? , 2010, International Journal of Computer Vision.

[9]  Robert Pless,et al.  Consistent Temporal Variations in Many Outdoor Scenes , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Andrew Jones,et al.  Digital Reunification of the Parthenon and its Sculptures , 2003, VAST.

[11]  Robert Pless,et al.  Toward Fully Automatic Geo-Location and Geo-Orientation of Static Outdoor Cameras , 2008, 2008 IEEE Workshop on Applications of Computer Vision.

[12]  Steven M. Seitz,et al.  Finding paths through the world's photos , 2008, SIGGRAPH 2008.

[13]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[14]  J. Michalsky,et al.  All-weather model for sky luminance distribution—Preliminary configuration and validation , 1993 .

[15]  Srinivasa G. Narasimhan,et al.  Clustering Appearance for Scene Analysis , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[16]  Robert Pless,et al.  Geolocating Static Cameras , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[17]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[18]  Alexei A. Efros,et al.  Photo clip art , 2007, SIGGRAPH 2007.

[19]  Wojciech Matusik,et al.  Factored time-lapse video , 2007, SIGGRAPH 2007.

[20]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[21]  Wojciech Matusik,et al.  What do color changes reveal about an outdoor scene? , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2007, SIGGRAPH 2007.

[23]  Hans-Peter Seidel,et al.  Relighting objects from image collections , 2009, CVPR 2009.

[24]  Alexei A. Efros,et al.  Geometric context from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[25]  Jian Sun,et al.  SkyFinder: attribute-based sky image search , 2009, SIGGRAPH 2009.

[26]  Paul Debevec Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography , 2008, SIGGRAPH Classes.

[27]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[28]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[29]  Michael Goesele,et al.  Multi-View Stereo for Community Photo Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[30]  Shree K. Nayar,et al.  Priors for Large Photo Collections and What They Reveal about Cameras , 2008, ECCV.

[31]  Ron O Dror,et al.  Statistical characterization of real-world illumination. , 2004, Journal of vision.

[32]  Peter Shirley,et al.  A practical analytic model for daylight , 1999, SIGGRAPH.

[33]  Heung-Yeung Shum,et al.  Radiometric calibration from a single image , 2004, CVPR 2004.

[34]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.