Fast Computation of Content-Sensitive Superpixels and Supervoxels Using Q-Distances

State-of-the-art researches model the data of images and videos as low-dimensional manifolds and generate superpixels/supervoxels in a content-sensitive way, which is achieved by computing geodesic centroidal Voronoi tessellation (GCVT) on manifolds. However, computing exact GCVTs is slow due to computationally expensive geodesic distances. In this paper, we propose a much faster queue-based graph distance (called q-distance). Our key idea is that for manifold regions in which q-distances are different from geodesic distances, GCVT is prone to placing more generators in them, and therefore after few iterations, the q-distance-induced tessellation is an exact GCVT. This idea works well in practice and we also prove it theoretically under moderate assumption. Our method is simple and easy to implement. It runs 6-8 times faster than state-of-the-art GCVT computation, and has an optimal approximation ratio O(1) and a linear time complexity O(N) for N-pixel images or N-voxel videos. A thorough evaluation of 31 superpixel methods on five image datasets and 8 supervoxel methods on four video datasets shows that our method consistently achieves the best over-segmentation accuracy. We also demonstrate the advantage of our method on one image and two video applications.

[1]  Xiaochun Cao,et al.  Regularity Preserved Superpixels and Supervoxels , 2014, IEEE Transactions on Multimedia.

[2]  Sven J. Dickinson,et al.  TurboPixels: Fast Superpixels Using Geometric Flows , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Peer Neubert,et al.  Compact Watershed and Preemptive SLIC: On Improving Trade-offs of Superpixel Segmentation Algorithms , 2014, 2014 22nd International Conference on Pattern Recognition.

[4]  Alan L. Yuille,et al.  Efficient Multilevel Brain Tumor Segmentation With Integrated Bayesian Model Classification , 2008, IEEE Transactions on Medical Imaging.

[5]  Jason J. Corso,et al.  Propagating multi-class pixel labels throughout video frames , 2010, 2010 Western New York Image Processing Workshop.

[6]  James M. Rehg,et al.  Video Segmentation by Tracking Many Figure-Ground Segments , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Sven J. Dickinson,et al.  Optimal Image and Video Closure by Superpixel Grouping , 2012, International Journal of Computer Vision.

[8]  Cevahir Çigla,et al.  Convexity constrained efficient superpixel and supervoxel extraction , 2015, Signal Process. Image Commun..

[9]  John W. Fisher,et al.  A Video Representation Using Temporal Superpixels , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  James M. Rehg,et al.  The Middle Child Problem: Revisiting Parametric Min-Cut and Seeds for Object Proposals , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[11]  Jan Kautz,et al.  Superpixel Sampling Networks , 2018, ECCV.

[12]  Hongbin Zha,et al.  Structure-sensitive superpixels via geodesic distance , 2011, ICCV.

[13]  Daniel P. Huttenlocher,et al.  Efficient Graph-Based Image Segmentation , 2004, International Journal of Computer Vision.

[14]  Thomas Brox,et al.  A Unified Video Segmentation Benchmark: Annotation, Metrics and Analysis , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  B. S. Manjunath,et al.  Color image segmentation , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[17]  Rainer Stiefelhagen,et al.  An evaluation of the compactness of superpixels , 2014, Pattern Recognit. Lett..

[18]  Michael Beetz,et al.  Depth-adaptive superpixels , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[19]  Stefano Soatto,et al.  Quick Shift and Kernel Methods for Mode Seeking , 2008, ECCV.

[20]  Bastian Leibe,et al.  Superpixels: An evaluation of the state-of-the-art , 2016, Comput. Vis. Image Underst..

[21]  Paria Mehrani,et al.  Superpixels and Supervoxels in an Energy Optimization Framework , 2010, ECCV.

[22]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Stewart Burn,et al.  Superpixels via pseudo-Boolean optimization , 2011, 2011 International Conference on Computer Vision.

[24]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  Yong-Jin Liu,et al.  Manifold SLIC: A Fast Method to Compute Content-Sensitive Superpixels , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Wanda Benesova,et al.  Fast Superpixel Segmentation Using Morphological Processing , 2014 .

[28]  Jianxiong Xiao,et al.  SUN RGB-D: A RGB-D scene understanding benchmark suite , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Yong-Jin Liu,et al.  Content-Sensitive Supervoxels via Uniform Tessellations on Video Manifolds , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Zhengqin Li,et al.  Superpixel segmentation using Linear Spectral Clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Mei Han,et al.  Efficient hierarchical graph-based video segmentation , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Martial Hebert,et al.  Learning to Find Object Boundaries Using Motion Cues , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[33]  Rynson W. H. Lau,et al.  SuperCNN: A Superpixelwise Convolutional Neural Network for Salient Object Detection , 2015, International Journal of Computer Vision.

[34]  Leo Grady,et al.  Random Walks for Image Segmentation , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Ronen Basri,et al.  Hierarchy and adaptivity in segmenting visual scenes , 2006, Nature.

[36]  Florentin Wörgötter,et al.  Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Shimon Ullman,et al.  Class-Specific, Top-Down Segmentation , 2002, ECCV.

[38]  Luis E. Ortiz,et al.  Parsing clothing in fashion photographs , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Qiang Du,et al.  Centroidal Voronoi Tessellations: Applications and Algorithms , 1999, SIAM Rev..

[40]  Kaleem Siddiqi,et al.  Parts of visual form: computational aspects , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[42]  Jitendra Malik,et al.  Spectral grouping using the Nystrom method , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Cordelia Schmid,et al.  Learning object class detectors from weakly annotated video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Ronald L. Rivest,et al.  Introduction to Algorithms, third edition , 2009 .

[45]  Frédo Durand,et al.  A Topological Approach to Hierarchical Segmentation using Mean Shift , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Yong-Jin Liu,et al.  Intrinsic Manifold SLIC: A Simple and Efficient Method for Computing Content-Sensitive Superpixels , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Gernot A. Fink,et al.  On the Influence of Superpixel Methods for Image Parsing , 2015, VISAPP.

[48]  Ronen Basri,et al.  Fast multiscale image segmentation , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[49]  Roberto Cipolla,et al.  Segmentation and Recognition Using Structure from Motion Point Clouds , 2008, ECCV.

[50]  Dennis Wei A Constant-Factor Bi-Criteria Approximation Guarantee for k-means++ , 2016, NIPS.

[51]  Ankit Aggarwal,et al.  Adaptive Sampling for k-Means Clustering , 2009, APPROX-RANDOM.

[52]  Jitendra Malik,et al.  Efficient spatiotemporal grouping using the Nystrom method , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[53]  Jitendra Malik,et al.  Occlusion boundary detection and figure/ground assignment from optical flow , 2011, CVPR 2011.

[54]  Kristen Grauman,et al.  Supervoxel-Consistent Foreground Propagation in Video , 2014, ECCV.

[55]  John MacCormick,et al.  Fast superpixels for video analysis , 2009, 2009 Workshop on Motion and Video Computing (WMVC).

[56]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[57]  Chenliang Xu,et al.  LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing , 2015, International Journal of Computer Vision.

[58]  Xiaohu Guo,et al.  Anisotropic Superpixel Generation Based on Mahalanobis Distance , 2016, Comput. Graph. Forum.

[59]  Luc Van Gool,et al.  Superpixel meshes for fast edge-preserving surface reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[60]  Jitendra Malik,et al.  Learning a classification model for segmentation , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[61]  Sanja Fidler,et al.  Real-time coarse-to-fine topologically preserving segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[62]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[63]  Umar Mohammed,et al.  Superpixel lattices , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[64]  Jitendra Malik,et al.  Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation , 2015, International Journal of Computer Vision.

[65]  Luc Van Gool,et al.  SEEDS: Superpixels Extracted via Energy-Driven Sampling , 2012, ECCV.

[66]  Stephen Gould,et al.  Decomposing a scene into geometric and semantically consistent regions , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[67]  Jitendra Malik,et al.  Learning to detect natural image boundaries using local brightness, color, and texture cues , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Jie Wang,et al.  VCells: Simple and Efficient Superpixels Using Edge-Weighted Centroidal Voronoi Tessellations , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[69]  Yong-Jin Liu,et al.  Evaluation on the Compactness of Supervoxels , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[70]  Rudolf Mester,et al.  Multichannel Segmentation Using Contour Relaxation: Fast Super-Pixels and Temporal Propagation , 2011, SCIA.

[71]  Rama Chellappa,et al.  Entropy rate superpixel segmentation , 2011, CVPR 2011.

[72]  Jan Kautz,et al.  Learning Superpixels with Segmentation-Aware Affinity Loss , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.