Superpixels for image and video processing based on proximity-weighted patch matching

In this paper, a temporal superpixel algorithm using proximity-weighted patch matching (PPM) is proposed to yield temporally consistent superpixels for image and video processing. PPM estimates the motion vector of a superpixel robustly, by considering the patch matching distances of neighboring superpixels as well as the superpixel itself. In each frame, we initialize superpixels by transferring the superpixel labels of the previous frame using PPM motion vectors. Then, we update the superpixel labels of boundary pixels by minimizing a cost function, which is composed of feature distance, compactness, contour, and temporal consistency terms. Finally, we carry out superpixel splitting, merging, and relabeling to regularize superpixel sizes and correct inaccurate labels. Extensive experimental results confirm that the proposed algorithm outperforms the state-of-the-art conventional algorithms significantly. Also, it is demonstrated that the proposed algorithm can be applied to video object segmentation and video saliency detection tasks.

[1]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Hui Wang,et al.  Adaptive Nonlocal Random Walks for Image Superpixel Segmentation , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Ting Liu,et al.  SSHMT: Semi-supervised Hierarchical Merge Tree for Electron Microscopy Image Segmentation , 2016, ECCV.

[4]  Yong-Jin Liu,et al.  Intrinsic Manifold SLIC: A Simple and Efficient Method for Computing Content-Sensitive Superpixels , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  John W. Fisher,et al.  A Video Representation Using Temporal Superpixels , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Sanja Fidler,et al.  Real-time coarse-to-fine topologically preserving segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Youbao Tang,et al.  Saliency Detection via Combining Region-Level and Pixel-Level Predictions with CNNs , 2016, ECCV.

[8]  Jian Sun,et al.  Geodesic Saliency Using Background Priors , 2012, ECCV.

[9]  Chang-Su Kim,et al.  Sequential Clique Optimization for Video Object Segmentation , 2018, ECCV.

[10]  Luc Van Gool,et al.  SEEDS: Superpixels Extracted Via Energy-Driven Sampling , 2012, International Journal of Computer Vision.

[11]  Chang-Su Kim,et al.  Contour-Constrained Superpixels for Image and Video Processing , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Chang-Su Kim,et al.  Online Video Object Segmentation via Convolutional Trident Network , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Hong Qin,et al.  Video Saliency Detection via Spatial-Temporal Fusion and Low-Rank Coherency Diffusion , 2017, IEEE Transactions on Image Processing.

[14]  Bodo Rosenhahn,et al.  Occlusion-Aware Method for Temporally Consistent Superpixels , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Yong-Jin Liu,et al.  Manifold SLIC: A Fast Method to Compute Content-Sensitive Superpixels , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Bohyung Han,et al.  Superpixel-Based Tracking-by-Segmentation Using Markov Chains , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Zhengqin Li,et al.  Superpixel segmentation using Linear Spectral Clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Bodo Rosenhahn,et al.  Temporally Consistent Superpixels , 2013, 2013 IEEE International Conference on Computer Vision.

[20]  Yicong Zhou,et al.  Content-Adaptive Superpixel Segmentation , 2018, IEEE Transactions on Image Processing.

[21]  Sabine Süsstrunk,et al.  Superpixels and Polygons Using Simple Non-iterative Clustering , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Hong Qin,et al.  A Novel Bottom-Up Saliency Detection Method for Video With Dynamic Background , 2018, IEEE Signal Processing Letters.

[23]  Li Cao,et al.  Superpixel Segmentation Using Gaussian Mixture Model , 2016, IEEE Transactions on Image Processing.

[24]  Yihong Gong,et al.  Superpixel Hierarchy , 2016, IEEE Transactions on Image Processing.

[25]  Chenliang Xu,et al.  Streaming Hierarchical Video Segmentation , 2012, ECCV.

[26]  Chenliang Xu,et al.  Evaluation of super-voxel methods for early video processing , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Hong Qin,et al.  Bilevel Feature Learning for Video Saliency Detection , 2018, IEEE Transactions on Multimedia.

[28]  Alexandre X. Falcão,et al.  An Iterative Spanning Forest Framework for Superpixel Segmentation , 2018, IEEE Transactions on Image Processing.

[29]  Chang-Su Kim,et al.  Primary Object Segmentation in Videos Based on Region Augmentation and Reduction , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Vittorio Ferrari,et al.  Fast Object Segmentation in Unconstrained Video , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  Rainer Stiefelhagen,et al.  Measuring and evaluating the compactness of superpixels , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[32]  Chang-Su Kim,et al.  Primary Object Segmentation in Videos via Alternate Convex Optimization of Foreground and Background Distributions , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Xiaochun Cao,et al.  Regularity Preserved Superpixels and Supervoxels , 2014, IEEE Transactions on Multimedia.

[34]  Sven J. Dickinson,et al.  TurboPixels: Fast Superpixels Using Geometric Flows , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[35]  Jason J. Corso,et al.  Propagating multi-class pixel labels throughout video frames , 2010, 2010 Western New York Image Processing Workshop.

[36]  James M. Rehg,et al.  Motion Coherent Tracking with Multi-label MRF optimization , 2010, BMVC.

[37]  Frédo Durand,et al.  A Topological Approach to Hierarchical Segmentation using Mean Shift , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Hailin Jin,et al.  Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Akisato Kimura,et al.  Fully Automatic Extraction of Salient Objects from Videos in Near Real Time , 2010, Comput. J..

[40]  Hongbin Zha,et al.  Structure-sensitive superpixels via geodesic distance , 2011, ICCV.

[41]  Yunsong Li,et al.  Efficient Coarse-to-Fine Patch Match for Large Displacement Optical Flow , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Li Xu,et al.  Hierarchical Saliency Detection , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Jian Sun,et al.  Saliency Optimization from Robust Background Detection , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Chang-Su Kim,et al.  Temporal Superpixels Based on Proximity-Weighted Patch Matching , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[45]  Gilles Bertrand,et al.  Simple points, topological numbers and geodesic neighborhoods in cubic grids , 1994, Pattern Recognit. Lett..

[46]  Rama Chellappa,et al.  Entropy rate superpixel segmentation , 2011, CVPR 2011.

[47]  Geoffrey Egnal,et al.  Detecting Binocular Half-Occlusions: Empirical Comparisons of Five Approaches , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Chang-Su Kim,et al.  Spatiotemporal Saliency Detection for Video Sequences Based on Random Walk With Restart , 2015, IEEE Transactions on Image Processing.

[49]  Bastian Leibe,et al.  Superpixels: An evaluation of the state-of-the-art , 2016, Comput. Vis. Image Underst..

[50]  Umar Mohammed,et al.  Superpixel lattices , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Yunsong Li,et al.  Minimum barrier superpixel segmentation , 2018, Image Vis. Comput..

[52]  Santiago Manen,et al.  Online Video SEEDS for Temporal Window Objectness , 2013, 2013 IEEE International Conference on Computer Vision.

[53]  Chang-Su Kim,et al.  Semi-supervised Video Object Segmentation Using Multiple Random Walkers , 2016, BMVC.

[54]  Chang-Su Kim,et al.  Tracking-by-Segmentation Using Superpixel-Wise Neural Network , 2018, IEEE Access.

[55]  Cordelia Schmid,et al.  DeepFlow: Large Displacement Optical Flow with Deep Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[56]  Jitendra Malik,et al.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[57]  Junwei Han,et al.  DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Zhuowen Tu,et al.  Holistically-Nested Edge Detection , 2015, ICCV.

[59]  Hongbin Zha,et al.  Structure-Sensitive Superpixels via Geodesic Distance , 2011, 2011 International Conference on Computer Vision.

[60]  Jan Kautz,et al.  Superpixel Sampling Networks , 2018, ECCV.