Detection and segmentation of moving objects in complex scenes

In this paper, we address the difficult task of detecting and segmenting foreground moving objects in complex scenes. The sequences we consider exhibit highly dynamic backgrounds, illumination changes and low contrasts, and can have been shot by a moving camera. Three main steps compose the proposed method. First, a set of moving points is selected within a sub-grid of image pixels. A multi-cue descriptor is associated to each of these points. Clusters of points are then formed using a variable bandwidth mean shift technique with automatic bandwidth selection. Finally, segmentation of the object associated to a given cluster is performed using graph cuts. Experiments and comparisons to other motion detection methods on challenging sequences demonstrate the performance of the proposed method for video analysis in complex scenes.

[1]  David J. Fleet,et al.  Performance of optical flow techniques , 1994, International Journal of Computer Vision.

[2]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[3]  Larry S. Davis,et al.  Background Updating for Visual Surveillance , 2005, ISVC.

[4]  Ioannis Patras,et al.  Probabilistic Confidence Measures for Block Matching Motion Estimation , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[6]  Nikos Paragios,et al.  Motion-based background subtraction using adaptive kernel density estimation , 2004, CVPR 2004.

[7]  Shigeru Akamatsu,et al.  Comparative performance of different skin chrominance models and chrominance spaces for the automatic detection of human faces in color images , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[8]  Stan Sclaroff,et al.  Segmenting foreground objects from a dynamic textured background via a robust Kalman filter , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[9]  René Vidal,et al.  A Unified Algebraic Approach to 2-D and 3-D Motion Segmentation , 2004, ECCV.

[10]  Jitendra Malik,et al.  Robust Multiple Car Tracking with Occlusion Reasoning , 1994, ECCV.

[11]  Michael J. Black,et al.  A framework for the robust estimation of optical flow , 1993, 1993 (4th) International Conference on Computer Vision.

[12]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Matthew P. Wand,et al.  Kernel Smoothing , 1995 .

[14]  Nikos Komodakis,et al.  Fast, Approximately Optimal Solutions for Single and Dynamic MRFs , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Jitendra Malik,et al.  Robust computation of optical flow in a multi-scale differential framework , 2005, International Journal of Computer Vision.

[16]  Richard P. Wildes.  A measure of motion salience for surveillance applications , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[17]  Touradj Ebrahimi,et al.  Video object extraction based on adaptive background and statistical change detection , 2000, IS&T/SPIE Electronic Imaging.

[18]  Yair Weiss,et al.  Smoothness in layers: Motion segmentation using nonparametric mixture estimation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  René Vidal,et al.  A closed form solution to direct motion segmentation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[20]  M. Hazelton.  Variable kernel density estimation , 2003 .

[21]  Mubarak Shah,et al.  Accurate motion layer segmentation and matting , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Eero P. Simoncelli.  Distributed representation and analysis of visual motion , 1993 .

[24]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Jerry D. Gibson,et al.  Handbook of Image and Video Processing , 2000 .

[26]  Jian-xiong Dong,et al.  Fast SVM training algorithm with decomposition on very large data sets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  D. W. Scott,et al.  Variable Kernel Density Estimation , 1992 .

[28]  Michael J. Black,et al.  Mixture models for optical flow computation , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Qiong Liu,et al.  A robust skin color based face detection algorithm , 2010, 2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010).

[30]  Larry S. Davis,et al.  Non-parametric Model for Background Subtraction , 2000, ECCV.

[31]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[32]  E. S. Pearson,et al.  ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE PART I , 1928 .

[33]  Dorin Comaniciu,et al.  An Algorithm for Data-Driven Bandwidth Selection , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Yu-Jin Zhang,et al.  Advances in image and video segmentation , 2006 .

[35]  Steven S. Beauchemin,et al.  The computation of optical flow , 1995, CSUR.

[36]  Paul L. Rosin.  Thresholding for change detection , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[37]  Harpreet S. Sawhney,et al.  Compact Representations of Videos Through Dominant and Multiple Motion Estimation , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Stuart J. Russell,et al.  Image Segmentation in Video Sequences: A Probabilistic Approach , 1997, UAI.

[39]  Takeo Kanade,et al.  Advances in Cooperative Multi-Sensor Video Surveillance , 1999 .

[40]  Harpreet S. Sawhney,et al.  Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding , 1995, Proceedings of IEEE International Conference on Computer Vision.

[41]  J. Odobez,et al.  Separation of Moving Regions from Background in an Image Sequence Acquired with a Mobil Camera , 1997 .

[42]  Ramesh C. Jain,et al.  On the Analysis of Accumulative Difference Pictures from Image Sequences of Real World Scenes , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[44]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[45]  Daphne Koller,et al.  Learning Spatial Context: Using Stuff to Find Things , 2008, ECCV.

[46]  Stefano Soatto,et al.  Dynamic Textures , 2003, International Journal of Computer Vision.

[47]  Richa Singh,et al.  A Robust Skin Color Based Face Detection Algorithm , 2003 .

[48]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[49]  Rudolf Mester,et al.  Detection of moving objects using a robust displacement estimation including a statistical error analysis , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[50]  Sankar K. Pal,et al.  A review on image segmentation techniques , 1993, Pattern Recognit..

[51]  Michael J. Black,et al.  The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields , 1996, Comput. Vis. Image Underst..

[52]  William H. Press,et al.  Book-Review - Numerical Recipes in Pascal - the Art of Scientific Computing , 1989 .

[53]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Patrick Pérez,et al.  Track and Cut: Simultaneous Tracking and Segmentation of Multiple Objects with Graph Cuts , 2008, EURASIP J. Image Video Process..

[56]  Abdelkrim Meziane.  Digital images segmentation: a state of art of the different methods , 2002 .

[57]  Olga Veksler,et al.  Markov random fields with efficient approximations , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[58]  Patrick Pérez,et al.  Bandwidth selection for kernel estimation in mixed multi-dimensional spaces , 2007, ArXiv.

[59]  L. Wixson.  Detecting Salient Motion by Accumulating Directionally-Consistent Flow , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[60]  Georgios Tziritas,et al.  Adaptive detection and localization of moving objects in image sequences , 1999, Signal Process. Image Commun..

[61]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[62]  D. Greig,et al.  Exact Maximum A Posteriori Estimation for Binary Images , 1989 .

[63]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[64]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[65]  Shyjan Mahamud,et al.  Comparing Belief Propagation and Graph Cuts for Novelty Detection , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[66]  John R. Kender,et al.  Finding skin in color images , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[67]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[68]  Ilangko Balasingham,et al.  Are the Wavelet Transforms the Best Filter Banks for Image Compression? , 2008, EURASIP J. Image Video Process..

[69]  Patrick Bouthemy,et al.  A maximality principle applied to a contrario motion detection , 2005, IEEE International Conference on Image Processing 2005.

[70]  Kwang-Ting Cheng,et al.  Learning a sparse, corner-based representation for time-varying background modelling , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[71]  Vladimir Kolmogorov,et al.  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision , 2004, IEEE Trans. Pattern Anal. Mach. Intell..

[72]  J. Marron,et al.  Improved Variable Window Kernel Estimates of Probability Densities , 1995 .

[73]  Ben Kröse,et al.  Confidence measures for image motion estimation , 1997 .

[74]  Ying-li Tian,et al.  Robust Salient Motion Detection with Complex Background for Real-Time Video Surveillance , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[75]  Yaser Sheikh,et al.  Bayesian modeling of dynamic scenes for object detection , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[77]  Badrinath Roysam,et al.  Image change detection algorithms: a systematic survey , 2005, IEEE Transactions on Image Processing.

[78]  Patrick Bouthemy,et al.  Computation and analysis of image motion: A synopsis of current problems and methods , 1996, International Journal of Computer Vision.

[79]  Til Aach,et al.  Bayesian algorithms for adaptive change detection in image sequences using Markov random fields , 1995, Signal Process. Image Commun..

[80]  Andrew Blake,et al.  LogCut - Efficient Graph Cut Optimization for Markov Random Fields , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[81]  Jean-Marc Odobez,et al.  Robust Multiresolution Estimation of Parametric Motion Models , 1995, J. Vis. Commun. Image Represent..

[82]  C. Quesenberry,et al.  A nonparametric estimate of a multivariate density function , 1965 .

[83]  Shrinivas J. Pundlik,et al.  Motion Segmentation at Any Speed , 2006, BMVC.

[84]  Nikos Paragios,et al.  A MRF-based approach for real-time subway monitoring , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[85]  Chin-Seng Chua,et al.  Statistical background modeling for non-stationary camera , 2003, Pattern Recognit. Lett..

[86]  W. Eric L. Grimson,et al.  Using adaptive tracking to classify and monitor activities in a site , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[87]  Patrick Pérez,et al.  Joint Tracking and Segmentation of Objects Using Graph Cuts , 2007, ACIVS.

[88]  A. Pentland,et al.  Robust estimation of a multi-layered motion representation , 1991, Proceedings of the IEEE Workshop on Visual Motion.

[89]  Lucia Ballerini,et al.  Time-Varying Image Processing and Moving Object Recognition , 1997 .

[90]  Patrick Bouthemy,et al.  Moving object detection in color image sequences using region-level graph labeling , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[91]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[92]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[93]  Dorin Comaniciu,et al.  The Variable Bandwidth Mean Shift and Data-Driven Scale Selection , 2001, ICCV.

[94]  H. Niemann,et al.  Adaptive change detection for real-time surveillance applications , 2000, Proceedings Third IEEE International Workshop on Visual Surveillance.

[95]  Hans-Hellmut Nagel,et al.  New likelihood test methods for change detection in image sequences , 1984, Comput. Vis. Graph. Image Process..

[96]  Jean-Christophe Terrillon,et al.  Comparative Performance of Different Chrominance Spaces for Color Segmentation and Detection of Human Faces in Complex Scene Images , 1999 .