A Hole Filling Approach Based on Background Reconstruction for View Synthesis in 3D Video

The depth image based rendering (DIBR) plays a key role in 3D video synthesis, by which other virtual views can be generated from a 2D video and its depth map. However, in the synthesis process, the background occluded by the foreground objects might be exposed in the new view, resulting in some holes in the synthetized video. In this paper, a hole filling approach based on background reconstruction is proposed, in which the temporal correlation information in both the 2D video and its corresponding depth map are exploited to construct a background video. To construct a clean background video, the foreground objects are detected and removed. Also motion compensation is applied to make the background reconstruction model suitable for moving camera scenario. Each frame is projected to the current plane where a modified Gaussian mixture model is performed. The constructed background video is used to eliminate the holes in the synthetized video. Our experimental results have indicated that the proposed approach has better quality of the synthetized 3D video compared with the other methods.

[1]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[2]  Taali Martin,et al.  Smoothing depth maps for improved steroscopic image quality , 2004, SPIE Optics East.

[3]  Yu Huang,et al.  A layered method of visibility resolving in depth image-based rendering , 2008, 2008 19th International Conference on Pattern Recognition.

[4]  Liang-Gee Chen,et al.  Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[5]  Yao Zhao,et al.  Depth Map Driven Hole Filling Algorithm Exploiting Temporal Correlation Information , 2014, IEEE Transactions on Broadcasting.

[6]  Ludovic J. Angot,et al.  A 2D to 3D video and image conversion technique based on a bilateral filter , 2010, Electronic Imaging.

[7]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[8]  Toshiaki Fujii,et al.  View generation with 3D warping using depth information for FTV , 2009, Signal Process. Image Commun..

[9]  Dar-Shyang Lee,et al.  Effective Gaussian mixture learning for video background subtraction , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[11]  Ghassan Al-Regib,et al.  Hierarchical Hole-Filling For Depth-Based View Synthesis in FTV and 3D Video , 2012, IEEE Journal of Selected Topics in Signal Processing.

[12]  P. KaewTrakulPong,et al.  An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection , 2002 .

[13]  Oscar C. Au,et al.  Novel temporal domain hole filling based on background modeling for view synthesis , 2012, 2012 19th IEEE International Conference on Image Processing.

[14]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[15]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Christine Guillemot,et al.  Depth-based image completion for view synthesis , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[17]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[18]  Manoranjan Paul,et al.  Improved Gaussian mixtures for robust object detection by adaptive multi-background generation , 2008, 2008 19th International Conference on Pattern Recognition.

[19]  Thomas Wiegand,et al.  Temporally consistent handling of disocclusions with texture synthesis for depth-image-based rendering , 2010, 2010 IEEE International Conference on Image Processing.

[20]  Tian-Sheuan Chang,et al.  Stereoscopic images generation with directional Gaussian filter , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[21]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[22]  Hideo Saito,et al.  A Novel Inpainting-Based Layered Depth Video for 3DTV , 2011, IEEE Transactions on Broadcasting.

[23]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[24]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[25]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[26]  Kwanghoon Sohn,et al.  Space-Time Hole Filling With Random Walks in View Extrapolation for 3D Video , 2013, IEEE Transactions on Image Processing.

[27]  Changick Kim,et al.  A Novel Depth-Based Virtual View Synthesis Method for Free Viewpoint Video , 2013, IEEE Transactions on Broadcasting.

[28]  Pei-Jun Lee,et al.  Nongeometric Distortion Smoothing Approach for Depth Map Preprocessing , 2011, IEEE Transactions on Multimedia.

[29]  Andrew Blake,et al.  Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming , 2006, International Journal of Computer Vision.

[30]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[31]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.