Sports VR Content Generation from Regular Camera Feeds

With the recent availability of commodity Virtual Reality (VR) products, immersive video content is receiving a significant interest. However, producing high-quality VR content often requires upgrading the entire production pipeline, which is costly and time-consuming. In this work, we propose using video feeds from regular broadcasting cameras to generate immersive content. We utilize the motion of the main camera to generate a wide-angle panorama. Using various techniques, we remove the parallax and align all video feeds. We then overlay parts from each video feed on the main panorama using Poisson blending. We examined our technique on various sports including basketball, ice hockey and volleyball. Subjective studies show that most participants rated their immersive experience when viewing our generated content between Good to Excellent. In addition, most participants rated their sense of presence to be similar to ground-truth content captured using a GoPro Omni 360 camera rig.

[1]  Mohamed Hefeeda,et al.  Adaptive streaming of interactive free viewpoint videos to heterogeneous clients , 2016, MMSys.

[2]  David Salesin,et al.  Photographing long scenes with multi-viewpoint panoramas , 2006, ACM Trans. Graph..

[3]  Cheng-Hsin Hsu,et al.  Performance Measurements of Virtual Reality Systems: Quantifying the Timing and Positioning Accuracy , 2016, ACM Multimedia.

[4]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[5]  Masatoshi Okutomi,et al.  Seamless image cloning by a closed form solution of a modified Poisson problem , 2012, SA '12.

[6]  Touradj Ebrahimi,et al.  Multimodal Dataset for Assessment of Quality of Experience in Immersive Multimedia , 2015, ACM Multimedia.

[7]  Aljoscha Smolic,et al.  MasterCam FVV: Robust registration of multiview sports video to a static high-resolution master camera for free viewpoint video , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[8]  Carsten Griwodz,et al.  Processing Panorama Video in Real-time , 2014, Int. J. Semantic Comput..

[9]  Hujun Bao,et al.  Multi-Viewpoint Panorama Construction With Wide-Baseline Images , 2016, IEEE Transactions on Image Processing.

[10]  Tomás Pajdla,et al.  Multi-view reconstruction preserving weakly-supported surfaces , 2011, CVPR 2011.

[11]  Yaser Sheikh,et al.  Point-less calibration: Camera parameters from gradient-based alignment to edge images , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[12]  Peter Eisert,et al.  Creation of High-Resolution Video Panoramas of Sport Events , 2006, Eighth IEEE International Symposium on Multimedia (ISM'06).

[13]  Markus H. Gross,et al.  Panoramic Video from Unstructured Camera Arrays , 2015, Comput. Graph. Forum.

[14]  Yoichi Sato,et al.  Shape-Preserving Half-Projective Warps for Image Stitching , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Toshiaki Fujii,et al.  A practical implementation of free viewpoint video system for soccer games , 2015, Electronic Imaging.

[16]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[17]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.

[18]  Michael S. Brown,et al.  Constructing image panoramas using dual-homography warping , 2011, CVPR 2011.

[19]  Michael S. Brown,et al.  As-Projective-As-Possible Image Stitching with Moving DLT , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Cornelius Malerczyk,et al.  3D-Reconstruction of Soccer Scenes , 2007, 2007 3DTV Conference.

[21]  Carsten Griwodz,et al.  Soccer video and player position dataset , 2014, MMSys '14.

[22]  Miska M. Hannuksela,et al.  HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications , 2016, ACM Multimedia.

[23]  David Salesin,et al.  Photographing long scenes with multi-viewpoint panoramas , 2006, SIGGRAPH 2006.

[24]  Narendra Ahuja,et al.  ROBUST VIDEO REGISTRATION APPLIED TO FIELD-SPORTS VIDEO ANALYSIS , 2012 .

[25]  Jonathan T. Barron,et al.  Jump: virtual reality video , 2016, ACM Trans. Graph..

[26]  Holger Regenbrecht,et al.  PanoVC: Pervasive telepresence using mobile phones , 2016, 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[27]  Mohamed Hefeeda,et al.  Streaming Virtual Reality Content , 2016, ArXiv.

[28]  Wojciech Matusik,et al.  Gradient-based 2D-to-3D Conversion for Soccer Videos , 2015, ACM Multimedia.

[29]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[30]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[31]  Dieter Schmalstieg,et al.  Real-time panoramic mapping and tracking on mobile phones , 2010, 2010 IEEE Virtual Reality Conference (VR).

[32]  Markus H. Gross,et al.  Novel‐View Synthesis of Outdoor Sport Events Using an Adaptive View‐Dependent Geometry , 2012, Comput. Graph. Forum.