GPU-accelerated real-time free-viewpoint DIBR for 3DTV

Multi-view 3D video may succeed stereo 3DTV in multimedia and TV applications. The MPEG committee has installed a special task force to establish a standard for multiview 3D video coding. Enabling free-viewpoint selection by the consumer is one of the major purposes of multi-view 3D encoded data. One focal point of our research work concerns an efficient implementation of the rendering part of stereoscopic free-viewpoint visualization, which is a computationally expensive task and significantly determines the final image quality. The free-viewpoint DIBR algorithm that is described in this article is implemented with an off-theshelf GPU that can be integrated in advanced 3DTV systems. We present the principal steps of a representative freeviewpoint DIBR, based on image warping and image artifact reduction techniques, and show the key differences between the reference software and our GPU implementation. One of those differences is the joint execution of signal processing blocks to share memory usage. Using a combination of the highly parallel programming architecture CUDA and a graphics API, we have achieved real-time performance executing 1080p HD multi-view video with a rendering quality that is comparable to the software implementation.

[1]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[2]  Toshiaki Fujii,et al.  View generation with 3D warping using depth information for FTV , 2009, Signal Process. Image Commun..

[3]  Charlie C. L. Wang,et al.  Gradient based image completion by solving the Poisson equation , 2007, Comput. Graph..

[4]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[5]  Gauthier Lafruit,et al.  Real-time stereo-based view synthesis algorithms: A unified framework and evaluation on commodity GPUs , 2009, Signal Process. Image Commun..

[6]  Ricardo Marroquim,et al.  Introduction to GPU Programming with GLSL , 2009, 2009 Tutorials of the XXII Brazilian Symposium on Computer Graphics and Image Processing.

[7]  E. Bondarev,et al.  Performance-efficient architecture for free-viewpoint 3DTV receiver , 2010, 2010 Digest of Technical Papers International Conference on Consumer Electronics (ICCE).

[8]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[9]  C. Fehn,et al.  Interactive 3-DTV-Concepts and Key Technologies , 2006 .

[10]  In Kyu Park,et al.  Accelerating Depth Image-Based Rendering Using GPU , 2006, MRCS.

[11]  Yo-Sung Ho.  3DTV: Technical challenges for realistic experiences , 2013, 2013 20th International Conference on Systems, Signals and Image Processing (IWSSIP).

[12]  Manuel Menezes de Oliveira Neto,et al.  Relief texture mapping , 2000, SIGGRAPH.

[13]  Peter H. N. de With,et al.  Quality improving techniques for free-viewpoint DIBR , 2010, Electronic Imaging.

[14]  Wen-mei W. Hwu,et al.  Optimization principles and application performance evaluation of a multithreaded GPU using CUDA , 2008, PPoPP.

[15]  Jens H. Krüger,et al.  A Survey of General‐Purpose Computation on Graphics Hardware , 2007, Eurographics.

[16]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[17]  Peter H. N. de With,et al.  Real-time free-viewpoint DIBR on GPUs for large base-line multi-view 3DTV videos , 2011, 2011 Visual Communications and Image Processing (VCIP).

[18]  Jiangbo Lu,et al.  A Scalable End-to-End Optimized Real-Time Image-Based Rendering Framework on Graphics Hardware , 2008, 2008 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[19]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[20]  Svitlana Zinger,et al.  Real-time free-viewpoint DIBR on GPUs for 3DTV systems , 2011, 2011 IEEE International Conference on Consumer Electronics -Berlin (ICCE-Berlin).

[21]  Yo-Sung Ho,et al.  Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video , 2009, 2009 Picture Coding Symposium.

[22]  Gauthier Lafruit,et al.  Migrating real-time depth image-based rendering from traditional to next-gen GPGPU , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[23]  Aljoscha Smolic,et al.  Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems , 2008, 2008 15th IEEE International Conference on Image Processing.

[24]  Peter H. N. de With,et al.  View Interpolation for Medical Images on Autostereoscopic Displays , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[25]  Kwanghoon Sohn,et al.  2D/3D freeview video generation for 3DTV system , 2008, 2008 15th IEEE International Conference on Image Processing.

[26]  Aljoscha Smolic,et al.  A Flexible 3D TV System for Different Multi-Baseline Geometries , 2006, 2006 IEEE International Conference on Multimedia and Expo.

[27]  David Blythe.  The Direct3D 10 system , 2006, ACM Trans. Graph..