Coherent spatial and temporal occlusion generation

A vastly growing number of productions from the entertainment industry are aiming at 3D movie theatres. These productions use a two-view format, primarily intended for eye-wear assisted viewing in a well defined environment. To get this 3D content into the home environment, where a large variety of 3D viewing conditions exists (e.g different display sizes, display types, viewing distance), we need a flexible 3D format that can adjust the depth effect. Such a format is the image plus depth format in which a video frame is enriched with depth information of all pixels in the video. This format can be extended with an additional layer for occluded video and associated depth, that contains what is behind objects in the video. To produce 3D content in this extended format, one has to deduce what is behind objects. There are various axes along which this occluded data can be obtained. This paper presents a method to automatically detect and fill the occluded areas exploiting the temporal axis. To get visually pleasing results, it is of utmost importance to make the inpainting globally consistent. To do so, we start by analyzing data along the temporal axis and compute a confidence for each pixel. Then pixels from the future and the past that are not visible in the current frame are weighted and accumulated based on computed confidences. These results are then fed to a generic multi-source framework that computes the occlusion layer based on the available confidences and occlusion data.

[1]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[2]  Anil Kokaram,et al.  Automated rig removal with bayesian motion interpolation , 2005 .

[3]  D. Nistér,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Harald Grossauer,et al.  A Combined PDE and Texture Synthesis Approach to Inpainting , 2004, ECCV.

[5]  R. Klein Gunnewiek,et al.  Robust image, depth, and occlusion generation from uncalibrated stereo , 2008, Electronic Imaging.

[6]  Bart Barenbrug.  3 D Throughout the video chain , 2006 .

[7]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[8]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[9]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[10]  Guillermo Sapiro,et al.  Simultaneous structure and texture image inpainting , 2003, IEEE Trans. Image Process..

[11]  Mei Han,et al.  Multiple Motion Scene Reconstruction with Uncalibrated Cameras , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[12]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[13]  Wolfgang Effelsberg,et al.  Robust background estimation for complex video sequences , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[14]  Guillermo Sapiro,et al.  Navier-stokes, fluid dynamics, and image and video inpainting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[15]  Bart Gerard Bernard Barenbrug.  Declipse 2: multi-layer image and depth with transparency made practical , 2009, Electronic Imaging.

[16]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[17]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[18]  Manuel Menezes de Oliveira Neto,et al.  Fast Digital Image Inpainting , 2001, VIIP.

[19]  Antonio Torralba,et al.  Depth Estimation from Image Structure , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Chris Varekamp,et al.  Enabling Introduction of Stereoscopic (3D) Video: Formats and Compression Standards , 2007, 2007 IEEE International Conference on Image Processing.

[21]  Chi-Keung Tang,et al.  Image repairing: robust image synthesis by adaptive ND tensor voting , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[22]  Ruigang Yang,et al.  Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation and Occlusion Handling , 2006, CVPR.