A layer extraction system based on dominant motion estimation and global registration

We describe a system that extracts layers from a video sequence based on a method estimating and stabilizing the dominant motion. Our approach performs pair-wise registration of images parameterized by 2D affine or projective transformation as the first step. This pair-wise registration is based on: (1) hierarchical parameter estimation and refinement; (2) feature-matching; (3) FFT (fast Fourier transformation)-based global matching; and (4) RANSAC-based parameter estimation. A global registration is then performed among overlapping multi-frames. The global registration is based on inferring the topology (i.e. spatio-temporal relation among frames) and the characterization of the likelihood within overlapping areas. As the last step, we extract the background layer that consists of static components of the video stream after motion estimation. Our background extraction is based on deriving a pixel-wise color distribution in time and provides a basis for a compact description of a video. The presented approach is illustrated by a set of challenging examples.

[1]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[2]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[3]  Rama Chellappa,et al.  Evaluation of image stabilization algorithms , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[4]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[6]  Gérard G. Medioni,et al.  A graph-based global registration for 2D mosaics , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[7]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Anil K. Jain,et al.  A background model initialization algorithm for video surveillance , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[9]  B. N. Chatterji,et al.  An FFT-based technique for translation, rotation, and scale-invariant image registration , 1996, IEEE Trans. Image Process..

[10]  P. Anandan,et al.  Efficient representations of video sequences and their applications , 1996, Signal Process. Image Commun..

[11]  P. Debevec,et al.  Image-based modeling, rendering, and lighting , 2002, IEEE Computer Graphics and Applications.

[12]  Leonard McMillan,et al.  Plenoptic Modeling: An Image-Based Rendering System , 2023 .

[13]  Harpreet S. Sawhney,et al.  True Multi-Image Alignment and Its Application to Mosaicing and Lens Distortion Correction , 1999, IEEE Trans. Pattern Anal. Mach. Intell..