Saliency Aggregation: A Data-Driven Approach

A variety of methods have been developed for visual saliency analysis. These methods often complement each other. This paper addresses the problem of aggregating various saliency analysis methods such that the aggregation result outperforms each individual one. We have two major observations. First, different methods perform differently in saliency analysis. Second, the performance of a saliency analysis method varies with individual images. Our idea is to use data-driven approaches to saliency aggregation that appropriately consider the performance gaps among individual methods and the performance dependence of each method on individual images. This paper discusses various data-driven approaches and finds that the image-dependent aggregation method works best. Specifically, our method uses a Conditional Random Field (CRF) framework for saliency aggregation that not only models the contribution from individual saliency map but also the interaction between neighboring pixels. To account for the dependence of aggregation on an individual image, our approach selects a subset of images similar to the input image from a training data set and trains the CRF aggregation model only using this subset instead of the whole training set. Our experiments on public saliency benchmarks show that our aggregation method outperforms each individual saliency method and is robust with the selection of aggregated methods.

[1]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2015, IEEE Transactions on Image Processing.

[2]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[3]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[4]  Ali Borji,et al.  Boosting bottom-up and top-down visual features for saliency estimation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Jian Sun,et al.  Geodesic Saliency Using Background Priors , 2012, ECCV.

[6]  Ming-Hsuan Yang,et al.  Top-down visual saliency via joint CRF and dictionary learning , 2012, CVPR.

[7]  C. Koch,et al.  A saliency-based search mechanism for overt and covert shifts of visual attention , 2000, Vision Research.

[8]  Gabriela Csurka,et al.  A framework for visual saliency detection with applications to image thumbnailing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[9]  Jian Sun,et al.  Salient object detection by composition , 2011, 2011 International Conference on Computer Vision.

[10]  Hongbin Zha,et al.  Salient object detection for searched web images via global saliency , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  Xueqing Li,et al.  Warp propagation for video resizing , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[12]  Yoichi Sato,et al.  Appearance-Based Gaze Estimation Using Visual Saliency , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[14]  Rongrong Ji,et al.  What are we looking for: Towards statistical modeling of saccadic eye movements and visual saliency , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[16]  Mubarak Shah,et al.  Visual attention detection in video sequences using spatiotemporal cues , 2006, MM '06.

[17]  Naila Murray,et al.  Saliency estimation using a non-parametric low-level vision model , 2011, CVPR 2011.

[18]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[19]  Cordelia Schmid,et al.  Discriminative spatial saliency for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Liqing Zhang,et al.  Saliency Detection: A Spectral Residual Approach , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Zhaoping Li A saliency map in primary visual cortex , 2002, Trends in Cognitive Sciences.

[22]  Touradj Ebrahimi,et al.  The JPEG2000 still image coding system: an overview , 2000, IEEE Trans. Consumer Electron..

[23]  C. Koch,et al.  Computational modelling of visual attention , 2001, Nature Reviews Neuroscience.

[24]  Ying Wu,et al.  A unified approach to salient object detection via low rank matrix recovery , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[26]  Martial Hebert,et al.  Segmentation of Salient Regions in Outdoor Scenes Using Imagery and 3-D Data , 2008, 2008 IEEE Workshop on Applications of Computer Vision.

[27]  Nuno Vasconcelos,et al.  Spatiotemporal Saliency in Dynamic Scenes , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Meng Wang,et al.  Image saliency: From intrinsic to extrinsic context , 2011, CVPR 2011.

[29]  Ronen Basri,et al.  Inferring region salience from binary and gray-level images , 2003, Pattern Recognit..

[30]  Lihi Zelnik-Manor,et al.  Saliency for image manipulation , 2013, The Visual Computer.

[31]  Antonio Torralba,et al.  Contextual Influences on Saliency , 2004 .

[32]  Harish Katti,et al.  Depth Matters: Influence of Depth Cues on Visual Saliency , 2012, ECCV.

[33]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[34]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[35]  HongJiang Zhang,et al.  Contrast-based image attention analysis by using fuzzy growing , 2003, MULTIMEDIA '03.

[36]  Deepu Rajan,et al.  Random walks on graphs to model saliency in images , 2009, CVPR.

[37]  Nuno Vasconcelos,et al.  On the plausibility of the discriminant center-surround hypothesis for visual saliency. , 2008, Journal of vision.

[38]  Sabine Süsstrunk,et al.  Salient Region Detection and Segmentation , 2008, ICVS.

[39]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Xing Xie,et al.  Effective browsing of web image search results , 2004, MIR '04.

[41]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[42]  Xueqing Li,et al.  Leveraging stereopsis for saliency analysis , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Lihi Zelnik-Manor,et al.  Context-Aware Saliency Detection , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  H. Nothdurft Salience from feature contrast: additivity across dimensions , 2000, Vision Research.

[45]  Yael Pritch,et al.  Saliency filters: Contrast based filtering for salient region detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.