Jump: virtual reality video

We present Jump, a practical system for capturing high resolution, omnidirectional stereo (ODS) video suitable for wide scale consumption in currently available virtual reality (VR) headsets. Our system consists of a video camera built using off-the-shelf components and a fully automatic stitching pipeline capable of capturing video content in the ODS format. We have discovered and analyzed the distortions inherent to ODS when used for VR display as well as those introduced by our capture method and show that they are small enough to make this approach suitable for capturing a wide variety of scenes. Our stitching algorithm produces robust results by reducing the problem to one of pairwise image interpolation followed by compositing. We introduce novel optical flow and compositing methods designed specifically for this task. Our algorithm is temporally coherent and efficient, is currently running at scale on a distributed computing platform, and is capable of processing hours of footage each day.

[1]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[2]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[3]  Tom Duff,et al.  Compositing digital images , 1984, SIGGRAPH.

[4]  Hiroshi Ishiguro,et al.  Omni-directional stereo for making global map , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[5]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[6]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.

[7]  Shree K. Nayar,et al.  Real-Time Omnidirectional and Panoramic Stereo , 1998 .

[8]  Harry Shum,et al.  Rendering with concentric mosaics , 1999, SIGGRAPH.

[9]  Naokazu Yokoya,et al.  Construction of an immersive mixed environment using an omnidirectional stereo image sensor , 2000, Proceedings IEEE Workshop on Omnidirectional Vision (Cat. No.PR00704).

[10]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[11]  Yael Pritch,et al.  Omnistereo: Panoramic Stereo Imaging , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Matthias Zwicker,et al.  Surface splatting , 2001, SIGGRAPH.

[13]  Leonard McMillan,et al.  A Real-Time Distributed Light Field Camera , 2002, Rendering Techniques.

[14]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[15]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[16]  Neil A. Dodgson,et al.  Variation and extrema of human interpupillary distance , 2004, IS&T/SPIE Electronic Imaging.

[17]  Damin Qin,et al.  Measurement for the Panum’s Fusional Area in Retinal Fovea Using a Three-Dimention Display Device , 2004 .

[18]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[19]  Susumu Tachi,et al.  TORNADO: omnistereo video imaging with rotating optics , 2005, IEEE Transactions on Visualization and Computer Graphics.

[20]  Markus Gross,et al.  Point-Based Graphics , 2007 .

[21]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[22]  Shmuel Peleg,et al.  Minimal Aspect Distortion (MAD) Mosaicing of Long Scenes , 2008, International Journal of Computer Vision.

[23]  Michael S. Langer,et al.  Analysis of disparity distortions in omnistereoscopic displays , 2010, TAP.

[24]  J. P. Lewis.  Fast Normalized Cross-Correlation , 2010 .

[25]  Michael F. Cohen,et al.  A Viewer-Centric Editor for 3D Movies , 2011, IEEE Computer Graphics and Applications.

[26]  Markus H. Gross,et al.  Mixed-order compositing for 3D paintings , 2011, SA '11.

[27]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Michael S. Langer,et al.  Panoramic stereo video textures , 2011, 2011 International Conference on Computer Vision.

[29]  Aljoscha Smolic,et al.  3D video and free viewpoint video - From capture to display , 2011, Pattern Recognit..

[30]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Vladlen Koltun,et al.  Efficient Nonlocal Regularization for Optical Flow , 2012, ECCV.

[32]  Frédo Durand,et al.  Decoupling algorithms from schedules for easy optimization of image processing pipelines , 2012, ACM Trans. Graph..

[33]  Peter Eisert,et al.  The Ultimate Immersive Experience: Panoramic 3D Video Acquisition , 2012, MMM.

[34]  Markus Gross,et al.  Practical temporal consistency for image-based graphics applications , 2012, ACM Trans. Graph..

[35]  Yael Pritch,et al.  Megastereo: Constructing High-Resolution Stereo Panoramas , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Adrien Bousseau,et al.  How do people edit light fields? , 2014, ACM Trans. Graph..

[37]  Aljoscha Smolic,et al.  Suplemental Material for Temporally Coherent Local Tone Mapping of HDR Video , 2014 .

[38]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[39]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jonathan T. Barron,et al.  Burst photography for high dynamic range and low-light imaging on mobile cameras , 2016, ACM Trans. Graph..

[42]  Jonathan T. Barron,et al.  The Fast Bilateral Solver , 2015, ECCV.

[43]  Luc Van Gool,et al.  Fast Optical Flow Using Dense Inverse Search , 2016, ECCV.

[44]  Christian Theobalt,et al.  Live intrinsic video , 2016, ACM Trans. Graph..