Using Photographs to Enhance Videos of a Static Scene

We present a framework for automatically enhancing videos of a static scene using a few photographs of the same scene. For example, our system can transfer photographic qualities such as high resolution, high dynamic range and better lighting from the photographs to the video. Additionally, the user can quickly modify the video by editing only a few still images of the scene. Finally, our system allows a user to remove unwanted objects and camera shake from the video. These capabilities are enabled by two technical contributions presented in this paper. First, we make several improvements to a state-of-the-art multiview stereo algorithm in order to compute view-dependent depths using video, photographs, and structure-from-motion data. Second, we present a novel image-based rendering algorithm that can re-render the input video using the appearance of the photographs while preserving certain temporal dynamics such as specularities and dynamic scene lighting.

[1]  Kiriakos N. Kutulakos,et al.  Plenoptic Image Editing , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[2]  P. Torr Geometric motion segmentation and model selection , 1998, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[3]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[4]  Reinhard Koch,et al.  Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera , 1999, DAGM-Symposium.

[5]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[6]  Keith J. Hanna,et al.  Hybrid stereo camera: an IBR approach for synthesis of very high resolution stereoscopic image sequences , 2001, SIGGRAPH.

[7]  Michael Bosse,et al.  Non-metric image-based rendering for video stabilization , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[8]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[9]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[11]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[12]  Irfan A. Essa,et al.  Learning video processing by example , 2002, Object recognition supported by user interaction for service robots.

[13]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[14]  Andrew Blake,et al.  Super-resolution Enhancement of Video , 2003, AISTATS.

[15]  Richard Szeliski,et al.  High dynamic range video , 2003, ACM Trans. Graph..

[16]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[17]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[18]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[19]  Ramesh Raskar,et al.  Image fusion for context enhancement and video surrealism , 2004, NPAR '04.

[20]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[21]  Serge J. Belongie,et al.  A Feature-Based Approach for Determining Dense Long Range Correspondences , 2004, ECCV.

[22]  Richard Szeliski,et al.  Extracting View-Dependent Depth Maps from a Collection of Images , 2004, International Journal of Computer Vision.

[23]  Harry Shum,et al.  Bayesian Correction of Image Intensity with Spatial Consideration , 2004, ECCV.

[24]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[25]  N. Ahuja,et al.  Seamless video editing , 2004, ICPR 2004.

[26]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[27]  Seth J. Teller,et al.  Video matching , 2004, Encyclopedia of Multimedia.

[28]  Seth Teller,et al.  Video matching , 2004, SIGGRAPH 2004.

[29]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[30]  Kiriakos N. Kutulakos,et al.  Plenoptic Image Editing , 2004, International Journal of Computer Vision.

[31]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[32]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, ACM Trans. Graph..

[33]  Brendan J. Frey,et al.  Video Epitomes , 2005, CVPR.

[34]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[35]  Leonard McMillan,et al.  Video enhancement using per-pixel virtual exposures , 2005, ACM Trans. Graph..

[36]  Yaron Caspi,et al.  Under the supervision of , 2003 .

[37]  Brendan J. Frey,et al.  Video Epitomes , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  David Nistér,et al.  Preemptive RANSAC for live structure and motion estimation , 2005, Machine Vision and Applications.

[39]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[40]  L. McMillan,et al.  Video enhancement using per-pixel virtual exposures , 2005, SIGGRAPH 2005.

[41]  Maneesh Agrawala,et al.  Interactive video cutout , 2005, SIGGRAPH 2005.

[42]  Maneesh Agrawala,et al.  Piecewise Image Registration in the Presence of Multiple Large Motions , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[43]  Ruigang Yang,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation and Occlusion Handling , 2006, CVPR.

[44]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[45]  R. Szeliski Locally adapted hierarchical basis preconditioning , 2006, SIGGRAPH 2006.

[46]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[47]  R. Szeliski Locally adapted hierarchical basis preconditioning , 2006, SIGGRAPH '06.

[48]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[49]  Sing Bing Kang,et al.  Stereo for Image-Based Rendering using Image Over-Segmentation , 2007, International Journal of Computer Vision.