Virtual Camera Synthesis for Soccer Game Replays

In this paper, we present a set of tools developed during the creation of a platform that allows the automatic generation of virtual views in a live soccer game production. Observing the scene through a multi-camera system, a 3D approximation of the players is computed and used for the synthesis of virtual views. The system is suitable both for static scenes, to create bullet time effects, and for video applications, where the virtual camera moves as the game plays.

[1]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[2]  Vladimir Kolmogorov,et al.  What energy functions can be minimized via graph cuts? , 2002, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[4]  M. Pollefeys,et al.  Unstructured video-based rendering: interactive exploration of casually captured videos , 2010, ACM Trans. Graph..

[5]  Olga Veksler,et al.  Markov random fields with efficient approximations , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[6]  Michael Goesele,et al.  Multi-View Stereo Revisited , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[7]  François Pitié,et al.  Automated colour grading using colour distribution transfer , 2007, Comput. Vis. Image Underst..

[8]  Takeo Kanade,et al.  Three-dimensional scene flow , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[10]  Yuichi Ohta,et al.  Live 3D Video in Soccer Stadium , 2003, SIGGRAPH '03.

[11]  Ruixia Xu,et al.  Convex programming colour constancy with a diagonal-offset model , 2005, IEEE International Conference on Image Processing 2005.

[12]  J. Starck,et al.  A Robust Free-Viewpoint Video System for Sport Scenes , 2007, 2007 3DTV Conference.

[13]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[15]  Hideo Saito,et al.  Immersive evaluation of virtualized soccer match at real stadium model , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[16]  Filiberto Pla,et al.  Affine Illumination Compensation for Multispectral Images , 2007, SCIA.

[17]  Ieee Xplore,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Information for Authors , 2022, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Andrew Blake,et al.  Probabilistic Fusion of Stereo with Color and Contrast for Bilayer Segmentation , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[20]  Montse Pardàs,et al.  Shape from incomplete silhouettes based on the reprojection error , 2010, Image Vis. Comput..

[21]  Robert T. Collins,et al.  A space-sweep approach to true multi-image matching , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Leif Kobbelt,et al.  A Surface-Growing Approach to Multi-View Stereo Reconstruction , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Yizhou Yu,et al.  Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping , 1998, Rendering Techniques.

[24]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[25]  Oliver Grau,et al.  A Bayesian Framework for Simultaneous Matting and 3D Reconstruction , 2007, Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007).

[26]  Hongbin Zha,et al.  Camera calibration from a circle and a coplanar point at infinity with applications to sports scenes analyses , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[27]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[28]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[29]  Pau Gargallo,et al.  Polyconvexification of the multi-label optical flow problem , 2010, 2010 IEEE International Conference on Image Processing.

[30]  Adrian Hilton,et al.  Surface Capture for Performance-Based Animation , 2007, IEEE Computer Graphics and Applications.

[31]  Yupin Luo,et al.  Automatic Camera Calibration for Images of Soccer Match , 2004, International Conference on Computational Intelligence.

[32]  Marcus A. Magnor,et al.  Virtual video camera: image-based viewpoint navigation through space and time , 2009, SIGGRAPH '09.

[33]  Mila Nikolova,et al.  Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models , 2006, SIAM J. Appl. Math..

[34]  Hideo Saito,et al.  Real-time video-based rendering from uncalibrated cameras using plane-sweep algorithm , 2010, J. Vis. Commun. Image Represent..

[35]  Pau Gargallo,et al.  A Narrow Band Method for the Convex Formulation of Discrete Multilabel Problems , 2010, Multiscale Model. Simul..

[36]  Edmond Boyer,et al.  Fusion of multiview silhouette cues using a space occupancy grid , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[37]  Jean-Yves Guillemaut,et al.  Robust graph-cut scene segmentation and reconstruction for free-viewpoint video of complex dynamic scenes , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[38]  A. Laurentini,et al.  The Visual Hull Concept for Silhouette-Based Image Understanding , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[39]  Montse Pardàs,et al.  Shape from inconsistent silhouette , 2008, Comput. Vis. Image Underst..

[40]  Christopher G. Harris,et al.  A Combined Corner and Edge Detector , 1988, Alvey Vision Conference.

[41]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.

[43]  Anil Kokaram,et al.  The linear Monge-Kantorovitch linear colour mapping for example-based colour transfer , 2007 .

[44]  Guillermo Sapiro,et al.  A Comprehensive Framework for Image Inpainting , 2010, IEEE Transactions on Image Processing.

[45]  Horst Bischof,et al.  A Globally Optimal Algorithm for Robust TV-L1 Range Image Integration , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[46]  Thomas W. Parks,et al.  Adaptive homogeneity-directed demosaicing algorithm , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[47]  Hongdong Li,et al.  Inverse tensor transfer with applications to novel view synthesis and multi-baseline stereo , 2006, Signal Process. Image Commun..

[48]  Bruce G. Baumgart,et al.  Geometric modeling for computer vision. , 1974 .

[49]  I. Kitahara,et al.  Live mixed-reality 3D video in soccer stadium , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[50]  Hideo Saito,et al.  Fly through view video generation of soccer scene , 2002, IWEC.

[51]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[52]  Takeo Kanade,et al.  Recording of multiple videos in a large-scale space for large-scale virtualized reality , 2001 .

[53]  Jean-Yves Guillemaut,et al.  Joint Multi-Layer Segmentation and Reconstruction for Free-Viewpoint Video Applications , 2011, International Journal of Computer Vision.

[54]  Marcelo Gattass,et al.  Automatic Camera Calibration for Image Sequences of a Football Match , 2001, ICAPR.

[55]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[56]  Vladimir Kolmogorov,et al.  Computing visual correspondence with occlusions using graph cuts , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[57]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[58]  Nicolas Papadakis,et al.  Multi-label Depth Estimation for Graph Cuts Stereo Problems , 2010, Journal of Mathematical Imaging and Vision.

[59]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[60]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[61]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[62]  Andrew Zisserman,et al.  Multiple view geometry in computer visiond , 2001 .

[63]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[64]  Marc Pollefeys,et al.  Unstructured video-based rendering: interactive exploration of casually captured videos , 2010, ACM Trans. Graph..

[65]  R. Zabih,et al.  Exact voxel occupancy with graph cuts , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[66]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..