3D-TV Production From Conventional Cameras for Sports Broadcast

3DTV production of live sports events presents a challenging problem involving conflicting requirements of maintaining broadcast stereo picture quality with practical problems in developing robust systems for cost effective deployment. In this paper we propose an alternative approach to stereo production in sports events using the conventional monocular broadcast cameras for 3D reconstruction of the event and subsequent stereo rendering. This approach has the potential advantage over stereo camera rigs of recovering full scene depth, allowing inter-ocular distance and convergence to be adapted according to the requirements of the target display and enabling stereo coverage from both existing and `virtual' camera positions without additional cameras. A prototype system is presented with results of sports TV production trials for rendering of stereo and free-viewpoint video sequences of soccer and rugby.

[1]  S. B. Kang,et al.  Survey of image-based representations and compression techniques , 2003, IEEE Trans. Circuits Syst. Video Technol..

[2]  Kwanghoon Sohn,et al.  A Stereoscopic Video Generation Method Using Stereoscopic Display Characterization and Motion Analysis , 2008, IEEE Transactions on Broadcasting.

[3]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Markus H. Gross,et al.  Articulated Billboards for Video‐based Rendering , 2010, Comput. Graph. Forum.

[5]  Adrian Hilton,et al.  Virtual view synthesis of people from multiple view video sequences , 2005, Graph. Model..

[6]  Cordelia Schmid,et al.  A performance evaluation of local descriptors , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Cordelia Schmid,et al.  A Comparison of Affine Region Detectors , 2005, International Journal of Computer Vision.

[8]  Cable Ieee Broadcast.  IEEE transactions on broadcasting , 1963 .

[9]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.

[10]  J. Starck,et al.  A Robust Free-Viewpoint Video System for Sport Scenes , 2007, 2007 3DTV Conference.

[11]  Oliver Grau,et al.  A Baysian Framework for Simultaneous Reconstruction and Matting , 2007 .

[12]  Michal Irani,et al.  Multi-Frame Correspondence Estimation Using Subspace Constraints , 2002, International Journal of Computer Vision.

[13]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Peter Hillman.  Segmentation of motion picture images and image sequences , 2002 .

[15]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[16]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[17]  Jean-Yves Guillemaut,et al.  Robust graph-cut scene segmentation and reconstruction for free-viewpoint video of complex dynamic scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  Ian D. Reid,et al.  A Multiple View Layered Representation for Dynamic Novel View Synthesis , 2003, BMVC.

[19]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[20]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Oliver Grau,et al.  Effects of camera aperture correction on keying of broadcast video , 2008 .

[22]  Miller,et al.  Interactive Free-Viewpoint Video , 2006 .

[23]  Tal Hassner,et al.  What Does the Scene Look Like from a Scene Point? , 2002, ECCV.

[24]  Harry Shum,et al.  Background Cut , 2006, ECCV.

[25]  K. Mueller,et al.  Development of a new MPEG standard for advanced 3D video applications , 2009, 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis.

[26]  Bruno Raffin,et al.  A Distributed Approach for Real Time 3D Modeling , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[27]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Ramesh C. Jain,et al.  Reality modeling and visualization from multiple video sequences , 1996, IEEE Computer Graphics and Applications.

[29]  Oliver Grau,et al.  A Bayesian Framework for Simultaneous Matting and 3D Reconstruction , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[30]  Edmond Boyer,et al.  Exact polyhedral visual hulls , 2003, BMVC.

[31]  Graham A. Thomas.  Real-time camera tracking using sports pitch markings , 2007, Journal of Real-Time Image Processing.

[32]  G. A. Thomas.  Real-Time Camera Pose Estimation for Augmenting Sports Scenes , 2006 .

[33]  L. Lipton.  Foundations of the Stereoscopic Cinema , 1982 .

[34]  Kiriakos N. Kutulakos.  Approximate N-View Stereo , 2000, ECCV.

[35]  Hideo Saito,et al.  Virtual Viewpoint Replay for a Soccer Match by View Interpolation From Multiple Cameras , 2007, IEEE Transactions on Multimedia.

[36]  Lance Williams,et al.  View Interpolation for Image Synthesis , 1993, SIGGRAPH.

[37]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[38]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[39]  Saied Moezzi,et al.  Virtual View Generation for 3D Digital Video , 1997, IEEE Multim..

[40]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[41]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Saito,et al.  Player viewpoint video synthesis using multiple cameras , 2006 .

[43]  Oliver Grau,et al.  Dual Mode Deformable Models for Free-Viewpoint Video of Outdoor Sports Events , 2007 .

[44]  Adrian Hilton,et al.  Proj ective Surface Refinement for Free-Viewpoint Video , 2006 .

[45]  Pushmeet Kohli,et al.  Reduce, reuse & recycle: Efficiently solving multi-label MRFs , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Tao Chen,et al.  3D-TV Content Storage and Transmission , 2011, IEEE Transactions on Broadcasting.