An end-to-end system for free viewpoint video for smooth camera transitions

In this paper, we present an end-to-end system for free viewpoint video for smooth camera transitions in sport scenes. Our system consists of a network of static computer vision cameras, a storage infrastructure and an interpolation rendering module, connected with a 10 Gigabit Ethernet network. The user of the system requests a viewpath for the virtual camera and the rendering module then generates the images using a depth-aware plane sweep approach. First, the foreground and background are separated and rendered independently. The foreground is rendered using a plane-sweep approach and the obtained depth map is split up in groups of players. Each group is assigned a global depth, which is used in a second plane sweep to restrict the depth. This will reduce artifacts, such as extra limbs and ghost players. The algorithm is demonstrated on actual soccer recordings. The system is fully automatic and can work in near real-time, thus providing virtual images of high quality in a fast manner.

[1]  Philippe Bekaert,et al.  Optimized Two-Party Video Chat with Restored Eye Contact Using Graphics Hardware , 2008, ICETE.

[2]  Markus H. Gross,et al.  Articulated Billboards for Video‐based Rendering , 2010, Comput. Graph. Forum.

[3]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[4]  Adrian Hilton,et al.  A Free-Viewpoint Video System for Visualization of Sport Scenes , 2007 .

[5]  Hideo Saito,et al.  Synthesizing Free-Viewpoing Images from Multiple View Videos in Soccer StadiumADIUM , 2006, International Conference on Computer Graphics, Imaging and Visualisation (CGIV'06).

[6]  Ruigang Yang,et al.  A Unified Approach To Real-Time, Multi-Resolution, Multi-Baseline 2d View Synthesis And 3d Depth Estimation Using Commodity Graphics Hardware , 2004, Int. J. Image Graph..

[7]  Gauthier Lafruit,et al.  Complexity Reduction of Real-time Depth Scanning on Graphics Hardware , 2009, VISAPP.

[8]  Henrique S. Malvar,et al.  High-quality linear interpolation for demosaicing of Bayer-patterned color images , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[9]  Oliver Grau,et al.  3D-TV Production From Conventional Cameras for Sports Broadcast , 2011, IEEE Transactions on Broadcasting.

[10]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[11]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[12]  Hideo Saito,et al.  Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras , 2007, IEEE Transactions on Multimedia.

[13]  Philippe Bekaert,et al.  Optimal Data Distribution for Versatile Finite Impulse Response Filtering on Next-Generation Graphics Hardware Using CUDA , 2009, 2009 15th International Conference on Parallel and Distributed Systems.

[14]  Ruigang Yang,et al.  Fast Image Segmentation and Smoothing Using Commodity Graphics Hardware , 2002, J. Graphics, GPU, & Game Tools.

[15]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[16]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[17]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[18]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[19]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[20]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[21]  Yuichi Ohta,et al.  Live 3D Video in Soccer Stadium , 2003, SIGGRAPH '03.

[22]  Philippe Bekaert,et al.  Raw Camera Image Demosaicing using Finite Impulse Response Filtering on Commodity GPU Hardware using CUDA , 2012, SIGMAP.