Light Field Video for Immersive Content Production

Light field video for content production is gaining both research and commercial interest as it has the potential to push the level of immersion for augmented and virtual reality to a close-to-reality experience. Light fields densely sample the viewing space of an object or scene using hundreds or even thousands of images with small displacements in between. However, a lack of standardised formats for compression, storage and transmission, along with the lack of tools to enable editing of light field data currently make it impractical for use in real-world content production. In this chapter we address two fundamental problems with light field data, namely representation and compression. Firstly we propose a method to obtain a 4D temporally coherent representation from the input light field video. This is an essential problem to solve that will enable efficient compression editing. Secondly, we present a method for compression of light field data based on the eigen texture method that provides a compact representation and enables efficient view-dependent rendering at interactive frame rates. These approaches achieve an order of magnitude compression and temporally consistent representation that are important steps towards practical toolsets for light field video content production.

[1]  Yael Moses,et al.  Multi-view Scene Flow Estimation: A View Centered Variational Approach , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Gavin S. P. Miller,et al.  Lazy Decompression of Surface Light Fields for Precomputed Global Illumination , 1998, Rendering Techniques.

[3]  Vagia Tsiminaki,et al.  Eigen Appearance Maps of Dynamic Shapes , 2016, ECCV.

[4]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Adrian Hilton,et al.  Semantically Coherent Co-Segmentation and Reconstruction of Dynamic Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Wei-Chao Chen,et al.  Light field mapping: efficient representation and hardware rendering of surface light fields , 2002, SIGGRAPH.

[7]  Katsushi Ikeuchi,et al.  Eigen-Texture Method: Appearance Compression and Synthesis Based on a 3D Model , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Marcus A. Magnor,et al.  Data compression for light-field rendering , 2000, IEEE Trans. Circuits Syst. Video Technol..

[9]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[10]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[11]  Takeo Kanade,et al.  Panoptic Studio: A Massively Multiview System for Social Motion Capture , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Changchang Wu,et al.  Towards Linear-Time Incremental Structure from Motion , 2013, 2013 International Conference on 3D Vision.

[13]  Jin Li,et al.  Compression of Lumigraph with multiple reference frame (MRF) prediction and just-in-time rendering , 2000, Proceedings DCC 2000. Data Compression Conference.

[14]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[15]  Bernd Girod,et al.  Light field compression using disparity-compensated lifting and shape adaptation , 2006, IEEE Transactions on Image Processing.

[16]  Adrian Hilton,et al.  4D Match Trees for Non-rigid Surface Alignment , 2016, ECCV.

[17]  Sven Wanner,et al.  Globally consistent depth labeling of 4D light fields , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Jean-Yves Guillemaut,et al.  Light Field Compression using Eigen Textures , 2019, 2019 International Conference on 3D Vision (3DV).

[19]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[20]  Gunnar Farnebäck,et al.  Two-Frame Motion Estimation Based on Polynomial Expansion , 2003, SCIA.

[21]  Georgios D. Evangelidis,et al.  Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[23]  Bruno Lévy,et al.  Least squares conformal maps for automatic texture atlas generation , 2002, ACM Trans. Graph..

[24]  Touradj Ebrahimi,et al.  Comparison and Evaluation of Light Field Image Coding Approaches , 2017, IEEE Journal of Selected Topics in Signal Processing.

[25]  Martin Kemp,et al.  Leonardo on Painting: An Anthology of Writings by Leonardo da Vinci; With a Selection of Documents Relating to his Career as an Artist , 1989 .

[26]  Frédo Durand,et al.  Unstructured Light Fields , 2012, Comput. Graph. Forum.

[27]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[28]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[29]  Paul E. Debevec,et al.  A system for acquiring, processing, and rendering panoramic light field stills for virtual reality , 2018, ACM Trans. Graph..

[30]  Simon Fuhrmann,et al.  MVE - A Multi-View Reconstruction Environment , 2014, GCH.

[31]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[32]  Sylvain Paris,et al.  SimpleFlow: A Non‐iterative, Sublinear Optical Flow Algorithm , 2012, Comput. Graph. Forum.

[33]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[34]  Cristian Sminchisescu,et al.  Large Displacement 3D Scene Flow with Occlusion Reasoning , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[35]  Kurt Keutzer,et al.  Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow , 2010, ECCV.

[36]  Daniel Cremers,et al.  Stereoscopic Scene Flow Computation for 3D Motion Understanding , 2011, International Journal of Computer Vision.

[37]  Jan-Michael Frahm,et al.  Sparse Dynamic 3D Reconstruction from Unsynchronized Videos , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[38]  Radu Bogdan Rusu,et al.  Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments , 2010, KI - Künstliche Intelligenz.

[39]  David Salesin,et al.  Surface light fields for 3D photography , 2000, SIGGRAPH.

[40]  Olga Sorkine-Hornung,et al.  Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction , 2016, ACM Trans. Graph..

[41]  Jean-Yves Guillemaut,et al.  4D Temporally Coherent Light-Field Video , 2017, 2017 International Conference on 3D Vision (3DV).

[42]  Ravi Ramamoorthi,et al.  Oriented Light-Field Windows for Scene Flow , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[43]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).