Learning Must-Link Constraints for Video Segmentation Based on Spectral Clustering

In recent years it has been shown that clustering and segmentation methods can greatly benefit from the integration of prior information in terms of must-link constraints. Very recently the use of such constraints has been integrated in a rigorous manner also in graph-based methods such as normalized cut. On the other hand spectral clustering as relaxation of the normalized cut has been shown to be among the best methods for video segmentation. In this paper we merge these two developments and propose to learn must-link constraints for video segmentation with spectral clustering. We show that the integration of learned must-link constraints not only improves the segmentation result but also significantly reduces the required runtime, making the use of costly spectral methods possible for today’s high quality video.

[1]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Patrick J. Flynn,et al.  The 20th Anniversary of the IEEE Transactions on Pattern Analysis and Machine Intelligence , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Claire Cardie,et al.  Proceedings of the Eighteenth International Conference on Machine Learning, 2001, p. 577–584. Constrained K-means Clustering with Background Knowledge , 2022 .

[4]  Jianbo Shi,et al.  Grouping with Bias , 2001, NIPS.

[5]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[6]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[7]  Charless C. Fowlkes,et al.  How Much Does Globalization Help Segmentation ? , 2004 .

[8]  Andrew Zisserman,et al.  Learning Layered Motion Segmentations of Video , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[9]  Brendan J. Frey,et al.  Generative Model for Layers of Appearance and Deformation , 2005, AISTATS.

[10]  Anders P. Eriksson,et al.  Normalized Cuts Revisited: A Reformulation for Segmentation with Linear Grouping Constraints , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[11]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[12]  Ian Davidson,et al.  Constrained Clustering: Advances in Algorithms, Theory, and Applications , 2008 .

[13]  Sylvain Paris,et al.  Edge-Preserving Smoothing and Mean-Shift Segmentation of Video Streams , 2008, ECCV.

[14]  Matthias Hein,et al.  Spectral clustering based on the graph p-Laplacian , 2009, ICML '09.

[15]  Dale Schuurmans,et al.  Fast normalized cut with linear constraints , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Xiaoou Tang,et al.  Constrained clustering via spectral regularization , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  William Brendel,et al.  Video object segmentation by tracking regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  Ian Davidson,et al.  Flexible constrained spectral clustering , 2010, KDD.

[19]  Charles A. Micchelli,et al.  On Spectral Learning , 2010, J. Mach. Learn. Res..

[20]  Jitendra Malik,et al.  Object Segmentation by Long Term Analysis of Point Trajectories , 2010, ECCV.

[21]  Eric L. Miller,et al.  Multiple Hypothesis Video Segmentation from Superpixel Flows , 2010, ECCV.

[22]  Matthias Hein,et al.  An Inverse Power Method for Nonlinear Eigenproblems with Applications in 1-Spectral Clustering and Sparse PCA , 2010, NIPS.

[23]  Mei Han,et al.  Efficient hierarchical graph-based video segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[24]  Ian Davidson,et al.  Active Spectral Clustering , 2010, 2010 IEEE International Conference on Data Mining.

[25]  Ivan Laptev,et al.  Track to the future: Spatio-temporal video segmentation with long-range motion cues , 2011, CVPR 2011.

[26]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Matthias Hein,et al.  Beyond Spectral Clustering - Tight Relaxations of Balanced Graph Cuts , 2011, NIPS.

[28]  Jitendra Malik,et al.  Occlusion boundary detection and figure/ground assignment from optical flow , 2011, CVPR 2011.

[29]  Roberto Cipolla,et al.  Spatio-temporal clustering of probabilistic region trajectories , 2011, 2011 International Conference on Computer Vision.

[30]  Yong Jae Lee,et al.  Key-segments for video object segmentation , 2011, 2011 International Conference on Computer Vision.

[31]  Nisheeth K. Vishnoi,et al.  Biased normalized cuts , 2011, CVPR 2011.

[32]  H. Sebastian Seung,et al.  Learning to Agglomerate Superpixel Hierarchies , 2011, NIPS.

[33]  Ullrich Köthe,et al.  Probabilistic image segmentation with closedness constraints , 2011, 2011 International Conference on Computer Vision.

[34]  Kurt Keutzer,et al.  Long term video segmentation through pixel level spectral clustering on GPUs , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[35]  Ronen Basri,et al.  Image Segmentation by Probabilistic Bottom-Up Aggregation and Cue Integration , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Narendra Ahuja,et al.  Exploiting nonlocal spatiotemporal structure for video segmentation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Bernt Schiele,et al.  Video Segmentation with Superpixels , 2012, ACCV.

[38]  Chenliang Xu,et al.  Evaluation of super-voxel methods for early video processing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Antonio Criminisi,et al.  Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning , 2012, Found. Trends Comput. Graph. Vis..

[40]  Matthias Hein,et al.  Constrained 1-Spectral Clustering , 2012, AISTATS.

[41]  Chenliang Xu,et al.  Streaming Hierarchical Video Segmentation , 2012, ECCV.

[42]  Katerina Fragkiadaki,et al.  Video segmentation by tracing discontinuities in a trajectory embedding , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Stella X. Yu,et al.  Progressive Multigrid Eigensolvers for Multiscale Spectral Segmentation , 2013, 2013 IEEE International Conference on Computer Vision.

[44]  James M. Rehg,et al.  Video Segmentation by Tracking Many Figure-Ground Segments , 2013, 2013 IEEE International Conference on Computer Vision.

[45]  Philippe Salembier,et al.  Hierarchical Video Representation with Trajectory Binary Partition Tree , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Thomas Brox,et al.  A Unified Video Segmentation Benchmark: Annotation, Metrics and Analysis , 2013, 2013 IEEE International Conference on Computer Vision.

[47]  Camillo J. Taylor,et al.  Towards Fast and Accurate Segmentation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[48]  Cristian Sminchisescu,et al.  Video Object Segmentation by Salient Segment Chain Composition , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[49]  John W. Fisher,et al.  A Video Representation Using Temporal Superpixels , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Thomas Brox,et al.  Spectral Graph Reduction for Efficient Image and Streaming Video Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Jitendra Malik,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Segmentation of Moving Objects by Long Term Video Analysis , 2022 .