Motion Trajectory Segmentation via Minimum Cost Multicuts

For the segmentation of moving objects in videos, the analysis of long-term point trajectories has been very popular recently. In this paper, we formulate the segmentation of a video sequence based on point trajectories as a minimum cost multicut problem. Unlike the commonly used spectral clustering formulation, the minimum cost multicut formulation gives natural rise to optimize not only for a cluster assignment but also for the number of clusters while allowing for varying cluster sizes. In this setup, we provide a method to create a long-term point trajectory graph with attractive and repulsive binary terms and outperform state-of-the-art methods based on spectral clustering on the FBMS-59 dataset and on the motion subtask of the VSB100 dataset.

[1]  Charless C. Fowlkes,et al.  Globally-optimal greedy algorithms for tracking a variable number of objects , 2011, CVPR 2011.

[2]  Ullrich Köthe,et al.  Cut, Glue, & Cut: A Fast, Approximate Solver for Multicut Partitioning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Jitendra Malik,et al.  Occlusion boundary detection and figure/ground assignment from optical flow , 2011, CVPR 2011.

[4]  Ivan Laptev,et al.  Track to the future: Spatio-temporal video segmentation with long-range motion cues , 2011, CVPR 2011.

[5]  Jitendra Malik,et al.  Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Gerhard Reinelt,et al.  Higher-order segmentation via multicuts , 2013, Comput. Vis. Image Underst..

[7]  Wei Wu,et al.  Robust Trajectory Clustering for Motion Segmentation , 2013, 2013 IEEE International Conference on Computer Vision.

[8]  Bernt Schiele,et al.  Learning Must-Link Constraints for Video Segmentation Based on Spectral Clustering , 2014, GCPR.

[9]  Ullrich Köthe,et al.  Globally Optimal Closed-Surface Segmentation for Connectomics , 2012, ECCV.

[10]  Michel Deza,et al.  Geometry of cuts and metrics , 2009, Algorithms and combinatorics.

[11]  S.,et al.  An Efficient Heuristic Procedure for Partitioning Graphs , 2022 .

[12]  Ullrich Köthe,et al.  Probabilistic image segmentation with closedness constraints , 2011, 2011 International Conference on Computer Vision.

[13]  Katerina Fragkiadaki,et al.  Two-Granularity Tracking: Mediating Trajectory and Detection Graphs for Tracking under Occlusions , 2012, ECCV.

[14]  Amos Fiat,et al.  Correlation clustering in general weighted graphs , 2006, Theor. Comput. Sci..

[15]  Avrim Blum,et al.  Correlation Clustering , 2004, Machine Learning.

[16]  Thomas Brox,et al.  Higher order motion models and spectral clustering , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[18]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Gerhard Reinelt,et al.  Globally Optimal Image Partitioning by Multicuts , 2011, EMMCVPR.

[20]  Pascal Fua,et al.  Tracking Interacting Objects Optimally Using Integer Programming , 2014, ECCV.

[21]  Jitendra Malik,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Segmentation of Moving Objects by Long Term Video Analysis , 2022 .

[22]  Bernt Schiele,et al.  Subgraph decomposition for multi-target tracking , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  René Vidal,et al.  Sparse subspace clustering , 2009, CVPR.

[24]  Cordelia Schmid,et al.  EpicFlow: Edge-preserving interpolation of correspondences for optical flow , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Katerina Fragkiadaki,et al.  Detection free tracking: Exploiting motion and topology for segmenting and tracking under entanglement , 2011, CVPR 2011.

[26]  Anil M. Cheriyadat,et al.  Non-negative matrix factorization of partial track data for motion segmentation , 2010, 2009 IEEE 12th International Conference on Computer Vision.

[27]  Marina Meila,et al.  Comparing Clusterings by the Variation of Information , 2003, COLT.

[28]  Jitendra Malik,et al.  Object Segmentation by Long Term Analysis of Point Trajectories , 2010, ECCV.

[29]  Poka-Yio.  Cut , 2015, Definitions.

[30]  Hongdong Li,et al.  Robust Motion Segmentation with Unknown Correspondences , 2014, ECCV.

[31]  M. R. Rao,et al.  The partition problem , 1993, Math. Program..

[32]  Michael Felsberg,et al.  Fast Segmentation of Sparse 3D Point Trajectories Using Group Theoretical Invariants , 2014, ACCV.

[33]  Bodo Rosenhahn,et al.  Multi-scale Clustering of Frame-to-Frame Correspondences for Motion Segmentation , 2012, ECCV.

[34]  Thomas Brox,et al.  A Unified Video Segmentation Benchmark: Annotation, Metrics and Analysis , 2013, 2013 IEEE International Conference on Computer Vision.

[35]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[36]  Zhuwen Li,et al.  Perspective Motion Segmentation via Collaborative Clustering , 2013, 2013 IEEE International Conference on Computer Vision.

[37]  Thomas Brox,et al.  Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions , 2011, 2011 International Conference on Computer Vision.

[38]  Jianbo Shi,et al.  Understanding popout through repulsion , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[39]  Thomas Brox,et al.  Efficient Decomposition of Image and Mesh Graphs by Lifted Multicuts , 2015, ICCV.

[40]  Thomas Brox,et al.  Spectral Graph Reduction for Efficient Image and Streaming Video Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Luc Van Gool,et al.  Motion Segmentation with Weak Labeling Priors , 2014, GCPR.

[42]  Bernt Schiele,et al.  Classifier based graph construction for video segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).