Depth Map Driven Hole Filling Algorithm Exploiting Temporal Correlation Information

The depth-image-based-rendering is a key technique to realize free viewpoint television. However, one critical problem in these systems is filling the disocclusion due to the 3-D warping process. This paper exploits the temporal correlation of texture and depth information to generate a background reference image. This is then used to fill the holes associated with the dynamic parts of the scene, whereas for static parts the traditional inpainting method is used. To generate the background reference image, the Gaussian mixture model is employed on the texture information, whereas, depth maps information are used to detect moving objects so as to enhance the background reference image. The proposed holes filling approach is particularly useful for the single-view-plus-depth format, where, contrary to the multi-view-plus-depth format, only information of one view could be used for this task. The experimental results show that objective and subjective gains can be achieved, and the gain ranges from 1 to 3 dB over the inpainting method.

[1]  Cable Ieee Broadcast.  IEEE transactions on broadcasting , 1963 .

[2]  Yao Zhao,et al.  View Synthesis Based on Background Update with Gaussian Mixture Model , 2012, PCM.

[3]  Liang Zhang,et al.  Stereoscopic image generation based on depth images for 3D TV , 2005, IEEE Transactions on Broadcasting.

[4]  Liang-Gee Chen,et al.  Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[5]  Yao Zhao,et al.  Macroblock Level Bits Allocation for Depth Maps in 3-D Video Coding , 2014, J. Signal Process. Syst..

[6]  Dar-Shyang Lee,et al.  Effective Gaussian mixture learning for video background subtraction , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  Shang-Hong Lai,et al.  Improved novel view synthesis from depth image with large baseline , 2008, 2008 19th International Conference on Pattern Recognition.

[8]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[9]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[10]  Masayuki Tanimoto,et al.  3D-TV System with Depth-Image-Based Rendering , 2012 .

[11]  Zhenzhong Chen,et al.  Depth No-Synthesis-Error Model for View Synthesis in 3-D Video , 2011, IEEE Transactions on Image Processing.

[12]  Zhenzhong Chen,et al.  Video Quality Assessment Based on Measuring Perceptual Noise From Spatial and Temporal Perspectives , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  Manoranjan Paul,et al.  Improved Gaussian mixtures for robust object detection by adaptive multi-background generation , 2008, 2008 19th International Conference on Pattern Recognition.

[14]  Ilangko Balasingham,et al.  Are the Wavelet Transforms the Best Filter Banks for Image Compression? , 2008, EURASIP J. Image Video Process..

[15]  Thomas Wiegand,et al.  Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering , 2010, 2010 IEEE International Conference on Image Processing.

[16]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[17]  Ismo Rakkolainen,et al.  A Survey of 3DTV Displays: Techniques and Technologies , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[18]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[19]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[20]  Zhi Liu,et al.  Low Complexity Depth Coding Assisted by Coding Information From Color Video , 2014, IEEE Transactions on Broadcasting.

[21]  Changick Kim,et al.  A Novel Depth-Based Virtual View Synthesis Method for Free Viewpoint Video , 2013, IEEE Transactions on Broadcasting.

[22]  Pei-Jun Lee,et al.  Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing , 2011, IEEE Transactions on Multimedia.

[23]  Demin Wang,et al.  Stereoscopic image generation based on depth images , 2004, 2004 International Conference on Image Processing, 2004. ICIP '04..

[24]  Dong Tian,et al.  Boundary Artifact Reduction in View Synthesis of 3D Video: From Perspective of Texture-Depth Alignment , 2011, IEEE Transactions on Broadcasting.

[25]  Liang-Gee Chen,et al.  Hybrid motion/depth-oriented inpainting for virtual view synthesis in multiview applications , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[26]  Yong Man Ro,et al.  Effect of Stimulus Width on the Perceived Visual Discomfort in Viewing Stereoscopic 3-D-TV , 2013, IEEE Transactions on Broadcasting.

[27]  P. KaewTrakulPong,et al.  An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection , 2002 .

[28]  Michael Schmeing,et al.  Depth Image Based Rendering: A faithful approach for the disocclusion problem , 2010, 2010 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[29]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[30]  Patrick Le Callet,et al.  Can 3D synthesized views be reliably assessed through usual subjective and objective evaluation protocols? , 2011, 2011 18th IEEE International Conference on Image Processing.