A Novel Depth-Based Virtual View Synthesis Method for Free Viewpoint Video

Free-viewpoint rendering (FVR) has become a popular topic in 3-D research. A promising technology in FVR is to generate virtual views using a single texture image and the corresponding depth image. A critical problem that occurs when generating virtual views is that the regions covered by the foreground objects in the original view may be disoccluded in the synthesized views. In this paper, a depth based disocclusion filling algorithm using patch-based texture synthesis is proposed. In contrast to the existing patch-based virtual view synthesis methods, the filling priority is driven by the robust structure tensor that efficiently reflects the overall structure of an image part and a new confidence term that produces fine synthesis results even near the foreground boundaries. Moreover, the best-matched patch is searched in the background regions and finally it is chosen through a new patch distance measure. Significant superiority of the proposed method over the state-of-the-art methods is presented by comparing the experimental results.

[1]  Patrick Pérez,et al.  Region filling and object removal by exemplar-based image inpainting , 2004, IEEE Transactions on Image Processing.

[2]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[3]  Hideo Saito,et al.  A Novel Inpainting-Based Layered Depth Video for 3DTV , 2011, IEEE Transactions on Broadcasting.

[4]  Oliver Schreer,et al.  3D Videocommunication: Algorithms, concepts and real-time systems in human centred communication , 2005 .

[5]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[6]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[7]  Tian-Sheuan Chang,et al.  Stereoscopic images generation with directional Gaussian filter , 2010, Proceedings of 2010 IEEE International Symposium on Circuits and Systems.

[8]  Liang-Gee Chen,et al.  Efficient Depth Image Based Rendering with Edge Dependent Depth Filter and Interpolation , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[9]  Michael Ashikhmin,et al.  Synthesizing natural textures , 2001, I3D '01.

[10]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[11]  Gabriel Peyré,et al.  Texture Synthesis with Grouplets , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[13]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[14]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[15]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[16]  Changick Kim,et al.  Depth-Based Disocclusion Filling for Virtual View Synthesis , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[17]  Marcelo Bertalmío,et al.  Strong-continuation, contrast-invariant inpainting with a third-order optimal PDE , 2006, IEEE Transactions on Image Processing.

[18]  Liang-Gee Chen,et al.  Real-Time Free Viewpoint Rendering System for Face-to-face Video Conference , 2008, 2008 Digest of Technical Papers - International Conference on Consumer Electronics.

[19]  Stefano Soatto,et al.  Dynamic Textures , 2003, International Journal of Computer Vision.

[20]  Guna Seetharaman,et al.  Application Of Papoulis-Gerchberg Method In Image Super-Resolution and Inpainting , 2009, Comput. J..

[21]  Leonard McMillan,et al.  Post-rendering 3D warping , 1997, SI3D.

[22]  Masayuki Tanimoto.  FTV (Free Viewpoint Television) for 3D Scene Reproduction and Creation , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[23]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[24]  Aljoscha Smolic,et al.  Reliability-based generation and view synthesis in layered depth video , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[25]  Djemel Ziou,et al.  A global approach for solving evolutive heat transfer for image denoising and inpainting , 2006, IEEE Transactions on Image Processing.

[26]  Aljoscha Smolic,et al.  Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems , 2008, 2008 15th IEEE International Conference on Image Processing.

[27]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[28]  Sylvain Lefebvre,et al.  Appearance-space texture synthesis , 2006, ACM Trans. Graph..

[29]  Christine Guillemot,et al.  Depth-based image completion for view synthesis , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[30]  Marc Levoy,et al.  Fast texture synthesis using tree-structured vector quantization , 2000, SIGGRAPH.

[31]  Taali Martin,et al.  Smoothing depth maps for improved steroscopic image quality , 2004, SPIE Optics East.