Panorama View With Spatiotemporal Occlusion Compensation for 3D Video Coding

The future of novel 3D display technologies largely depends on the design of efficient techniques for 3D video representation and coding. Recently, multiple view plus depth video formats have attracted many research efforts since they enable intermediate view estimation and permit to efficiently represent and compress 3D video sequences. In this paper, we present spatiotemporal occlusion compensation with panorama view (STOP), a novel 3D video coding technique based on the creation of a panorama view and occlusion coding in terms of spatiotemporal offsets. The panorama picture represents the most of the visual information acquired from multiple views using a single virtual view, characterized by a larger field of view. Encoding the panorama video with state-of-the-art HECV and representing occlusions with simple spatiotemporal ancillary information STOP achieves high-compression ratio and good visual quality with competitive results with respect to competing techniques. Moreover, STOP enables free viewpoint 3D TV applications whilst allowing legacy display to get a bidimensional service using a standard video codec and simple cropping operations.

[1]  Heiko Schwarz,et al.  3D High-Efficiency Video Coding for Multi-View Video and Depth Data , 2013, IEEE Transactions on Image Processing.

[2]  Liang Zhang,et al.  Stereoscopic image generation based on depth images for 3D TV , 2005, IEEE Transactions on Broadcasting.

[3]  Krzysztof Wegner,et al.  3D video compression by coding of disoccluded regions , 2012, 2012 19th IEEE International Conference on Image Processing.

[4]  Yo-Sung Ho,et al.  Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video , 2009, 2009 Picture Coding Symposium.

[5]  Panos Nasiopoulos,et al.  A low complexity mode decision approach for HEVC-based 3D video coding using a Bayesian method , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[6]  Sehoon Yea,et al.  View synthesis prediction for multiview video coding , 2009, Signal Process. Image Commun..

[7]  Pau Gargallo,et al.  Stereoscopic Image Inpainting: Distinct Depth Maps and Images Inpainting , 2010, 2010 20th International Conference on Pattern Recognition.

[8]  Josep R. Casas,et al.  Multi-View Video Representation Based on Fast Monte Carlo Surface Reconstruction , 2013, IEEE Transactions on Image Processing.

[9]  Alan C. Bovik,et al.  Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures , 2009, IEEE Signal Processing Magazine.

[10]  B. S. Manjunath,et al.  Improving the quality of depth image based rendering for 3D Video systems , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[11]  Aljoscha Smolic,et al.  An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution , 2009, 2009 Picture Coding Symposium.

[12]  Nam Ling,et al.  High Efficiency Video Coding and its 3D extension: A research perspective , 2012, 2012 7th IEEE Conference on Industrial Electronics and Applications (ICIEA).

[13]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[14]  Itu-T and Iso Iec Jtc.  Advanced video coding for generic audiovisual services , 2010 .

[15]  Aljoscha Smolic,et al.  Coding and intermediate view synthesis of multiview video plus depth , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[16]  Kuo-Chin Fan,et al.  Image Registration Using a New Edge-Based Approach , 1997, Comput. Vis. Image Underst..

[17]  Christine Guillemot,et al.  Depth-based image completion for view synthesis , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[18]  Gary J. Sullivan,et al.  Overview of the High Efficiency Video Coding (HEVC) Standard , 2012, IEEE Transactions on Circuits and Systems for Video Technology.

[19]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[20]  Tao Chen,et al.  3D-TV Content Storage and Transmission , 2011, IEEE Transactions on Broadcasting.

[21]  Michael Cohen,et al.  Rendering Layered Depth Images , 1997 .

[22]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[23]  T. Vlachos.  Cut detection in video sequences using phase correlation , 2000, IEEE Signal Processing Letters.

[24]  Thomas Wiegand,et al.  Depth Image-Based Rendering With Advanced Texture Synthesis for 3-D Video , 2010, IEEE Transactions on Multimedia.

[25]  Thomas Wiegand,et al.  Consistent spatio-temporal filling of disocclusions in the multiview-video-plus-depth format , 2012, 2012 IEEE 14th International Workshop on Multimedia Signal Processing (MMSP).

[26]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[28]  G. Bjontegaard,et al.  Calculation of Average PSNR Differences between RD-curves , 2001 .

[29]  Aljoscha Smolic,et al.  Efficient Prediction Structures for Multiview Video Coding , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[30]  Dongxiao Li,et al.  Depth-image-based rendering with spatial and temporal texture synthesis for 3DTV , 2013, EURASIP J. Image Video Process..

[31]  Thomas Wiegand,et al.  3D video formats and coding methods , 2010, 2010 IEEE International Conference on Image Processing.

[32]  Luis Salgado,et al.  Efficient spatio-temporal hole filling strategy for Kinect depth maps , 2012, Electronic Imaging.

[33]  Marco Grangetto,et al.  Depth image based rendering with inverse mapping , 2013, 2013 IEEE 15th International Workshop on Multimedia Signal Processing (MMSP).

[34]  Thomas Wiegand,et al.  Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering , 2010, 2010 IEEE International Conference on Image Processing.

[35]  Weisi Lin,et al.  Perceptual visual quality metrics: A survey , 2011, J. Vis. Commun. Image Represent..

[36]  Jaejoon Lee,et al.  3-D video coding using depth transition data , 2010, 28th Picture Coding Symposium.

[37]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[38]  Qiuwen Zhang,et al.  Fast Mode Decision for 3D-HEVC Depth Intracoding , 2014, TheScientificWorldJournal.

[39]  Thomas Wiegand,et al.  Depth image-based rendering with spatio-temporally consistent texture synthesis for 3-D video with global motion , 2012, 2012 19th IEEE International Conference on Image Processing.

[40]  Lisa M. Brown,et al.  A survey of image registration techniques , 1992, CSUR.

[41]  Jun Yu,et al.  An efficient method for scene cut detection , 2001, Pattern Recognit. Lett..

[42]  Shang-Hong Lai,et al.  Spatio-temporally Consistent Multi-view Video Synthesis for Autostereoscopic Displays , 2009, PCM.

[43]  Shang-Hong Lai,et al.  Global optimization for spatio-temporally consistent view synthesis , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[44]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[45]  Hui.  A Contour-Based Approach to Multisensor Image Registration , 1995 .

[46]  Marco Grangetto,et al.  Tile format: A novel frame compatible approach for 3D video broadcasting , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[47]  Chung-Lin Huang,et al.  A robust scene-change detection method for video segmentation , 2001, IEEE Trans. Circuits Syst. Video Technol..

[48]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[49]  Dongxiao Li,et al.  Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video , 2009, IEEE Transactions on Consumer Electronics.

[50]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[51]  BrownLisa Gottesfeld.  A survey of image registration techniques , 1992 .

[52]  Rik Van de Walle,et al.  3D video compression based on high efficiency video coding , 2012, IEEE Transactions on Consumer Electronics.

[53]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[54]  Thomas Sikora,et al.  Adaptive Image Warping for Hole Prevention in 3D View Synthesis , 2013, IEEE Transactions on Image Processing.

[55]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[56]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.