3D Object Transfer Between Non-Overlapping Videos

Given two video sequences of different scenes acquired with moving cameras, it is interesting to seamlessly transfer a 3D object from one sequence to the other. In this paper, we present a video-based approach to extract the alpha mattes of rigid or approximately rigid 3D objects from one or more source videos, and then geometrycorrectly transfer them into another target video of a different scene. Our framework builds upon techniques in camera pose estimation, 3D spatiotemporal video alignment, depth recovery, key-frame editing, natural video matting, and image-based rendering. Based on the explicit camera pose estimation, the camera trajectories of the source and target videos are aligned in 3D space. Combinied with the estimated dense depth information, this allows us to significantly relieve the burdens of key-frame editing and efficiently improve the quality of video matting. During the transfer, our approach not only correctly restores the geometric deformation of the 3D object due to the different camera trajectories, but also effectively retains the soft shadow and environmental lighting properties of the object to ensure that the augmenting object is in harmony with the target scene.

[1]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[2]  Mubarak Shah,et al.  Accurate motion layer segmentation and matting , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[3]  Jan Flusser,et al.  Image registration methods: a survey , 2003, Image Vis. Comput..

[4]  Andrew Fitzgibbon,et al.  Bayesian video matting using learnt image priors , 2004, CVPR 2004.

[5]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[6]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[7]  Andrew Zisserman,et al.  Combining scene and auto-calibration constraints , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[9]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[10]  B. Caprile,et al.  Using vanishing points for camera calibration , 1990, International Journal of Computer Vision.

[11]  David Salesin,et al.  Shadow matting and compositing , 2003, ACM Trans. Graph..

[12]  Seth Teller,et al.  Video matching , 2004, SIGGRAPH 2004.

[13]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[14]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[15]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[16]  Samuel Audet,et al.  Image-Based Rendering Using Image-Based Priors , 2006 .

[18]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[19]  David Salesin,et al.  Video matting of complex scenes , 2002, SIGGRAPH.

[20]  Yaron Caspi,et al.  Alignment of non-overlapping sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[21]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[22]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[23]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[24]  Tanveer F. Syeda-Mahmood,et al.  View-invariant alignment and matching of video sequences , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[25]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[26]  Mubarak Shah,et al.  Tri-view morphing , 2004, Comput. Vis. Image Underst..

[27]  M. Shah,et al.  Motion layer extraction in the presence of occlusion using graph cut , 2004, CVPR 2004.

[28]  Jiaya Jia,et al.  Poisson matting , 2004, SIGGRAPH 2004.

[29]  Carlo Tomasi,et al.  Alpha estimation in natural images , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[30]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[31]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..