Space-time scene manifolds

The space of images is known to be a nonlinear sub-space that is difficult to model. This paper derives an algorithm that walks within this space. We seek a manifold through the video volume that is constrained to lie locally in this space. Every local neighborhood within the manifold resembles some image patch. We call this the scene manifold because the solution traces the scene outline. For a broad class of inputs the problem can be posed as finding the shortest path in a graph and can thus be solved efficiently to produce the globally optimal solution. Constraining appearance rather than geometry gives rise to numerous new capabilities. Here we demonstrate the usefulness of this approach by posing the well-studied problem of mosaicing in a new way. Instead of treating it as geometrical alignment, we pose it as an appearance optimization. Since the manifold is constrained to lie in the space of valid image patches, the resulting mosaic is guaranteed to have the least distortions possible. Any small part of it can be seen in some image even though the manifold spans the whole video. Thus it can deal seamlessly with both static and dynamic scenes, with or without 3D parallax. Essentially, the method simultaneously solves two problems that have been solved only separately until now: alignment and mosaicing.

[1]  Shmuel Peleg,et al.  Mosaicing with Parallax using Time Warping , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[2]  Adam Finkelstein,et al.  Stylized video cubes , 2002, SCA '02.

[3]  Edsger W. Dijkstra,et al.  A note on two problems in connexion with graphs , 1959, Numerische Mathematik.

[4]  Daphna Weinshall,et al.  Mosaicing New Views: The Crossed-Slits Projection , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Shmuel Peleg,et al.  Panoramic mosaics by manifold projection , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Steven M. Seitz,et al.  The Space of All Stereo Images , 2004, International Journal of Computer Vision.

[7]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2005, International Journal of Computer Vision.

[8]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[9]  David Salesin,et al.  Panoramic video textures , 2005, ACM Trans. Graph..

[10]  Michal Irani,et al.  Recovery of Ego-Motion Using Region Alignment , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[11]  Eli Shechtman,et al.  Space-time video completion , 2004, CVPR 2004.

[12]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[13]  Andrew W. Fitzgibbon,et al.  Image-Based Rendering Using Image-Based Priors , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.