A Free-Viewpoint Video Renderer

Free-viewpoint video allows interactive control of the camera viewpoint in video playback using a three-dimensional (3D) scene representation. This paper describes the challenges and the state-of-the-art techniques adopted in developing a free-viewpoint renderer for 3D video production of people. The system requires video streams from a set of fixed or dynamic real-world video cameras and synthesizes arbitrary viewpoints on the GPU using a dynamic 3D scene model. We have released the renderer as an open source application for the research community (see the web information at the end of this paper). The renderer provides (i) the facility to simulate camera viewpoints from publicly available multiple-view video datasets and (ii) a baseline technique for free-viewpoint video synthesis in the development of interactive 3D video and 3DTV applications.

[1]  Wojciech Matusik,et al.  Articulated mesh animation from multi-view silhouettes , 2008, ACM Trans. Graph..

[2]  Oliver Grau,et al.  A combined studio production system for 3-D capturing of live action and immersive actor feedback , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Jérémie Allard,et al.  The GrImage Platform: A Mixed Reality Environment for Interactions , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[4]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[5]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[6]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[7]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[8]  Jean-Yves Guillemaut,et al.  Objective Quality Assessment in Free-Viewpoint Video Production , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[9]  Adrian Hilton,et al.  The Multiple-Camera 3-D Production Studio , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[11]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[12]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[13]  Kok-Lim Low,et al.  Blending multiple views , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[14]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[15]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[16]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[17]  Hans-Peter Seidel,et al.  Performance capture from sparse multi-view video , 2008, ACM Trans. Graph..

[18]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[19]  Adrian Hilton,et al.  Virtual view synthesis of people from multiple view video sequences , 2005, Graph. Model..

[20]  Yu Meng,et al.  Dmesh: Fast Depth-Image Meshing And Warping , 2004, Int. J. Image Graph..

[21]  Takeo Kanade,et al.  Image-based spatio-temporal modeling and view interpolation of dynamic events , 2005, TOGS.

[22]  Linda G. Shapiro,et al.  View-base Rendering: Visualizing Real Objects from Scanned Range and Color Data , 1997, Rendering Techniques.