Viewpoint-Free Photography for Virtual Reality

Viewpoint-free photography, i.e., interactively controlling the viewpoint of a photograph after capture, is a central challenge for real virtual reality (VR) experiences. In this chapter, we present algorithms that enable viewpoint-free photography from casual capture, i.e., footage easily captured with consumer cameras. We build on extensive work in image-based rendering, which often focuses on full or near-interpolation, where output viewpoints lie directly between captured images, or nearby. However, for 6-DOF VR experiences, it is essential to create viewpoint-free photos with a wide field-of-view and sufficient positional freedom to cover the range of motion a user might experience.

[1]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[2]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[3]  Reinhard Koch,et al.  Plenoptic Modeling and Rendering from Image Sequences Taken by Hand-Held Camera , 1999, DAGM-Symposium.

[4]  Paul E. Debevec,et al.  A system for acquiring, processing, and rendering panoramic light field stills for virtual reality , 2018, ACM Trans. Graph..

[5]  Yael Pritch,et al.  Megastereo: Constructing High-Resolution Stereo Panoramas , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Michael Goesele,et al.  Image-based rendering in the gradient domain , 2013, ACM Trans. Graph..

[7]  Minh N. Do,et al.  SEAGULL: Seam-Guided Local Alignment for Parallax-Tolerant Image Stitching , 2016, ECCV.

[8]  Fan Zhang,et al.  Parallax-Tolerant Image Stitching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Yael Pritch,et al.  Omnistereo: Panoramic Stereo Imaging , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Michael Goesele,et al.  Multi-View Stereo for Community Photo Collections , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[11]  Andrew W. Fitzgibbon,et al.  Fields of Experts for Image-based Rendering , 2006, BMVC.

[12]  David Salesin,et al.  Interactive digital photomontage , 2004, SIGGRAPH 2004.

[13]  Jian Sun,et al.  Guided Image Filtering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Andrew W. Fitzgibbon,et al.  Efficient new-view synthesis using pairwise dictionary priors , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  John Flynn,et al.  Deep Stereo: Learning to Predict New Views from the World's Imagery , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[17]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[18]  Carlos Hernandez,et al.  Multi-View Stereo: A Tutorial , 2015, Found. Trends Comput. Graph. Vis..

[19]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[20]  George Drettakis,et al.  A Bayesian Approach for Selective Image-Based Rendering Using Superpixels , 2015, 2015 International Conference on 3D Vision.

[21]  Jonathan T. Barron,et al.  Jump: virtual reality video , 2016, ACM Trans. Graph..

[22]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[23]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[24]  Torsten Sattler,et al.  A Multi-view Stereo Benchmark with High-Resolution Images and Multi-camera Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[26]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Hiroshi Ishiguro,et al.  Omni-directional stereo for making global map , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[28]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[29]  Daniel G. Aliaga,et al.  Sea of images , 2002 .

[30]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[31]  Markus H. Gross,et al.  Panoramic Video from Unstructured Camera Arrays , 2015, Comput. Graph. Forum.

[32]  M. Goesele,et al.  Floating scale surface reconstruction , 2014, ACM Trans. Graph..

[33]  Heiko Hirschmüller,et al.  Stereo Vision in Structured Environments by Consistent Semi-Global Matching , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[34]  Sing Bing Kang,et al.  Stereo for Image-Based Rendering using Image Over-Segmentation , 2007, International Journal of Computer Vision.

[35]  Adam Finkelstein,et al.  PatchMatch: a randomized correspondence algorithm for structural image editing , 2009, SIGGRAPH 2009.

[36]  Vladlen Koltun,et al.  Color map optimization for 3D reconstruction with consumer depth cameras , 2014, ACM Trans. Graph..

[37]  Simon J. Julier,et al.  Structured Prediction of Unobserved Voxels from a Single Depth Image , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  In-So Kweon,et al.  All-Around Depth from Small Motion with a Spherical Panoramic Camera , 2016, ECCV.

[39]  Richard Szeliski,et al.  Layered Depth Panoramas , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[41]  George Drettakis,et al.  Scalable inside-out image-based rendering , 2016, ACM Trans. Graph..

[42]  Minglun Gong,et al.  Full 3D reconstruction of transparent objects , 2018, ACM Trans. Graph..

[43]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[45]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[46]  Gordon Wetzstein,et al.  SpinVR: towards live-streaming 3D virtual reality video , 2017, ACM Trans. Graph..

[47]  Li Zhang,et al.  Soft 3D reconstruction for view synthesis , 2017, ACM Trans. Graph..

[48]  Andrew Fitzgibbon,et al.  Fast Image-based Rendering using Hierarchical Image-based Priors , 2005 .

[49]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[50]  Jan-Michael Frahm,et al.  Pixelwise View Selection for Unstructured Multi-View Stereo , 2016, ECCV.

[51]  Frédo Durand,et al.  Unstructured Light Fields , 2012, Comput. Graph. Forum.

[52]  Michael Goesele,et al.  Let There Be Color! Large-Scale Texturing of 3D Reconstructions , 2014, ECCV.

[53]  Ravi Ramamoorthi,et al.  Learning to Synthesize a 4D RGBD Light Field from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[54]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2005, International Journal of Computer Vision.

[55]  Jean-Philippe Pons,et al.  Efficient Multi-View Reconstruction of Large-Scale Scenes using Interest Points, Delaunay Triangulation and Graph Cuts , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[56]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[57]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Simon Fuhrmann,et al.  MVE - A Multi-View Reconstruction Environment , 2014, GCH.

[59]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[60]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[61]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Michael Garland,et al.  Surface simplification using quadric error metrics , 1997, SIGGRAPH.

[63]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[64]  George Drettakis,et al.  Depth synthesis and local warps for plausible image-based navigation , 2013, TOGS.

[65]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[66]  Andrew W. Fitzgibbon,et al.  Fast Image-based Rendering using Hierarchical Texture Priors , 2005, BMVC.

[67]  ARNO KNAPITSCH,et al.  Tanks and temples , 2017, ACM Trans. Graph..

[68]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[69]  Olga Sorkine-Hornung,et al.  Point Cloud Noise and Outlier Removal for Image-Based 3D Reconstruction , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[70]  In-So Kweon,et al.  High-Quality Depth from Uncalibrated Small Motion Clip , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Thomas Brox,et al.  Global, Dense Multiscale Reconstruction for a Billion Points , 2015, ICCV.

[72]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[73]  Carsten Rother,et al.  Fast Cost-Volume Filtering for Visual Correspondence and Beyond , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[74]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[75]  Christian Richardt,et al.  MegaParallax: Casual 360° Panoramas with Motion Parallax , 2019, IEEE Transactions on Visualization and Computer Graphics.

[76]  Richard Szeliski,et al.  Piecewise planar stereo for image-based rendering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[77]  Daniel Cremers,et al.  A Super-Resolution Framework for High-Accuracy Multiview Reconstruction , 2013, International Journal of Computer Vision.

[78]  Mark Meyer,et al.  Kernel-predicting convolutional networks for denoising Monte Carlo renderings , 2017, ACM Trans. Graph..