Depth image based compositing for stereo 3D

Compositing is an essential tool in modern film making. Established workflows exist for single-view video compositing, however new problems that demand new solutions arise when considering stereoscopic compositing. We investigate new methods for compositing live action stereo 3D, given two stereo camera systems and a corresponding depth map. We break the process up into three main steps; First, we show how to use trifocal tensors to robustly project 3D content from one video into another. Second, we analyze different image-based rendering methods for drawing objects to be composited into the new video. Finally, we describe a novel super-pixel based depth-test that increases robustness to errors in depth map accuracy. A user study was conducted to validate different steps of this process.

[1]  Raanan Fattal,et al.  Video stabilization using epipolar geometry , 2012, TOGS.

[2]  M. Gross,et al.  Nonlinear disparity mapping for stereoscopic 3D , 2010, ACM Trans. Graph..

[3]  Peter H. N. de With,et al.  Free-viewpoint depth image based rendering , 2010, J. Vis. Commun. Image Represent..

[4]  Peter Kauff,et al.  Production Rules for Stereo Acquisition , 2011, Proceedings of the IEEE.

[5]  Jonathan Starck.  Stereo in post-production , 2010, 3DVP '10.

[6]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[7]  Zeev Farbman,et al.  Coordinates for instant image cloning , 2009, ACM Trans. Graph..

[8]  Rafael Monroy,et al.  Disparity-Aware Stereo 3D Production Tools , 2011, 2011 Conference for Visual Media Production.

[9]  H. Opower.  Multiple view geometry in computer vision , 2002 .

[10]  Peter H. N. de With,et al.  Quality improving techniques for free-viewpoint DIBR , 2010, Electronic Imaging.

[11]  Matthias Zwicker,et al.  Stereoscopic 3D copy & paste , 2010, ACM Trans. Graph..

[12]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[13]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[14]  Christoph Fehn,et al.  Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV , 2004, IS&T/SPIE Electronic Imaging.

[15]  Markus H. Gross,et al.  StereoBrush: interactive 2D to 3D conversion using discontinuous warps , 2011, SBIM '11.

[16]  Nancy S. Pollard,et al.  Local layering , 2009, SIGGRAPH 2009.