Free-viewpoint video synthesis from mixed resolution multi-view images and low resolution depth maps

Streaming application of multi-view and free-viewpoint video is potentially attractive but due to the limitation of bandwidth, transmitting all multi-view video in high resolution may not be feasible. Our goal is to propose a new streaming data format that can be adapted to the limited bandwidth and capable of free-viewpoint video streaming using multi-view video plus depth (MVD). Given a requested free-viewpoint, we use the two closest views and corresponding depth maps to perform free-viewpoint video synthesis. We propose a new data format that consists of all views and corresponding depth maps in a lowered resolution, and the two closest views to the requested viewpoint in the high resolution. When the requested viewpoint changes, the two closest viewpoints will change, but one or both of views are transmitted only in the low resolution during the delay time. Therefore, the resolution compensation is required. In this paper, we investigated several cases where one or both of the views are transmitted only in the low resolution. We proposed adequate view synthesis method for multi resolution multi-view video plus depth. Experimental results show that our framework achieves view synthesis quality close to high resolution multi-view video plus depth.

[1]  Toshiaki Fujii,et al.  FTV for 3-D Spatial Communication , 2012, Proceedings of the IEEE.

[2]  S. Min,et al.  3D/2D convertible projection-type integral imaging using concave half mirror array. , 2010, Optics express.

[3]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[4]  Toshiaki Fujii,et al.  Free-Viewpoint TV , 2011, IEEE Signal Processing Magazine.

[5]  Peter Kauff,et al.  Three-Dimensional Video Postproduction and Processing , 2011, Proceedings of the IEEE.

[6]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[7]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Toshiaki Fujii,et al.  View Generation with 3D Warping Using Depth Information for FTV , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[9]  Toshiaki Fujii,et al.  The Seelinder: Cylindrical 3D display viewable from 360 degrees , 2010, J. Vis. Commun. Image Represent..

[10]  Toshiaki Fujii,et al.  Probabilistic reliability based view synthesis for FTV , 2010, 2010 IEEE International Conference on Image Processing.

[11]  Yasuhiro Takaki,et al.  Multi-projection of lenticular displays to construct a 256-view super multi-view display. , 2010, Optics express.

[12]  Ajay Luthra,et al.  Overview of the H.264/AVC video coding standard , 2003, IEEE Trans. Circuits Syst. Video Technol..

[13]  Juyong Park,et al.  32.1: Optimal Projector Configuration Design for 300‐Mpixel Light‐Field 3D Display , 2013 .

[14]  Toshiaki Fujii,et al.  Depth Coding using Downsampling and View based Upsampling , 2011 .

[15]  Shunsuke Yoshida.  fVisiOn: glasses-free tabletop 3D display to provide virtual 3D media naturally alongside real media , 2012, Defense, Security, and Sensing.

[16]  Masayuki Tanimoto,et al.  Multiview Imaging and 3DTV , 2007, IEEE Signal Processing Magazine.

[17]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[18]  Makoto Okui,et al.  Integral Three-Dimensional Television Using a 33-Megapixel Imaging System , 2010, Journal of Display Technology.

[19]  Toshiaki Fujii,et al.  Depth Coding using Downsampling and View based Upsampling , 2011 .

[20]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.