Joint projection filling method for occlusion handling in Depth-Image-Based Rendering

This paper addresses the disocclusion problem which may occur when using Depth-Image-Based Rendering (DIBR) techniques in 3DTV and Free-Viewpoint TV applications. A new DIBR technique is proposed, which combines three methods: a Joint Projection Filling (JPF) method to handle disocclusions in synthesized depth maps; a backward projection to synthesize virtual views; and a full-Z depth-aided inpainting to fill in disoccluded areas in textures. The JPF method performs the pixels warping for virtual depth map synthesis while making use of an occlusion-compatible pixel ordering strategy, to detect cracks and disocclusions, and to select the pixels to be propagated in the occlusion areas filling process. The full-Z depth-aided inpainting method fills in disocclusions with textures at the correct depth, preserving the boundaries of the objects. Ghosting artifacts, which might otherwise result from pixel projections, are here avoided by introducing a confidence measure on background pixels to be used in the JPF process.

[1]  Leonard McMillan,et al.  A List-Priority Rendering Algorithm for Redisplaying Projected Surfaces , 1995 .

[2]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[3]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[4]  Harry Shum,et al.  Review of image-based rendering techniques , 2000, Visual Communications and Image Processing.

[5]  Matthias Zwicker,et al.  Surfels: surface elements as rendering primitives , 2000, SIGGRAPH.

[6]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[7]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[8]  Matthias Zwicker,et al.  3 Ideal Resampling 3 . 1 Sampling and Aliasing , 2022 .

[9]  Patrick Pérez,et al.  Object removal by exemplar-based inpainting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[10]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[11]  Alexandru Telea,et al.  An Image Inpainting Technique Based on the Fast Marching Method , 2004, J. Graphics, GPU, & Game Tools.

[12]  Tsuhan Chen,et al.  A survey on image-based rendering - representation, sampling and compression , 2004, Signal Process. Image Commun..

[13]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[14]  Horst Bischof,et al.  A Duality Based Approach for Realtime TV-L1 Optical Flow , 2007, DAGM-Symposium.

[15]  Michael F. Cohen,et al.  Image and Video Matting: A Survey , 2007, Found. Trends Comput. Graph. Vis..

[16]  N. Atzpadin,et al.  Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability , 2007, Signal Process. Image Commun..

[17]  Ze-Nian Li,et al.  Review and Preview: Disocclusion by Inpainting for Image-Based Rendering , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[18]  Yo-Sung Ho,et al.  A Framework for Representation and Processing of Multi-view Video Using the Concept of Layered Depth Image , 2007, J. VLSI Signal Process..

[19]  Heung-Yeung Shum,et al.  Image-Based Rendering and Synthesis , 2007, IEEE Signal Processing Magazine.

[20]  Toshiaki Fujii,et al.  View generation with 3D warping using depth information for FTV , 2009, Signal Process. Image Commun..

[21]  Aljoscha Smolic,et al.  View Synthesis for Advanced 3D Video Systems , 2008, EURASIP J. Image Video Process..

[22]  Aljoscha Smolic,et al.  Reliability-based generation and view synthesis in layered depth video , 2008, 2008 IEEE 10th Workshop on Multimedia Signal Processing.

[23]  Yo-Sung Ho,et al.  Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video , 2009, 2009 Picture Coding Symposium.

[24]  Peter H. N. de With,et al.  Quality improving techniques in DIBR for free-viewpoint video , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[25]  Minh N. Do,et al.  Depth image-based rendering from multiple cameras with 3D propagation algorithm , 2009, IMMERSCOM.

[26]  A. Hilton,et al.  Wide-Baseline Matte Propagation for Indoor Scenes , 2009, 2009 Conference for Visual Media Production.

[27]  Christine Guillemot,et al.  Incremental-LDI for multi-view coding , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[28]  Béatrice Pesquet-Popescu,et al.  Depth-aided image inpainting for novel view synthesis , 2010, 2010 IEEE International Workshop on Multimedia Signal Processing.

[29]  Gaël Sourimant,et al.  Depth maps estimation and use for 3DTV , 2010 .

[30]  Christine Guillemot,et al.  Depth-based image completion for view synthesis , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).