Motion based decompositing of video

We present a method to decompose video sequences into layers that represent the relative depths of complex scenes. Our method combines spatial information with temporal occlusions to determine relative depths of these layers. Spatial information is obtained through edge detection and a customized contour completion algorithm. Activity in a scene is used to extract temporal occlusion events, which are in turn, used to classify objects as occluders or occludes. The path traversed by the moving objects determines the segmentation of the scene. Several examples of decompositing and compositing of video are shown. This approach can be applied in the pre-processing of sequences for compositing or tracking purposes and to determine the approximate 3D structure of a scene.

[1]  Ken Nakayama,et al.  Biological image motion processing: A review , 1985, Vision Research.

[2]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[3]  Stan Sclaroff,et al.  Active blobs , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[4]  Thomas S. Huang,et al.  Image processing , 1971 .

[5]  W. Eric L. Grimson,et al.  Using adaptive tracking to classify and monitor activities in a site , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[6]  G. Kanizsa,et al.  Organization in Vision: Essays on Gestalt Perception , 1979 .

[7]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[8]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[9]  Richard Szeliski,et al.  A layered approach to stereo reconstruction , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[10]  Ken-ichi Anjyo,et al.  Tour into the picture: using a spidery mesh interface to make animation from a single image , 1997, SIGGRAPH.

[11]  Brian G. Schunck,et al.  Interpolating cubic spline contours by minimizing second derivative discontinuity , 1990, [1990] Proceedings Third International Conference on Computer Vision.

[12]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Mei Han,et al.  Interactive 3D Modeling from Multiple Images Using Scene Regularities , 1998, SMILE.

[14]  James W. Davis,et al.  An appearance-based representation of action , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[15]  A. Pentland,et al.  Robust estimation of a multi-layered motion representation , 1991, Proceedings of the IEEE Workshop on Visual Motion.