Exploring visual attention and saliency modeling for task-based visual analysis

Abstract Memory, visual attention and perception play a critical role in the design of visualizations. The way users observe a visualization is affected by salient stimuli in a scene as well as by domain knowledge, interest, and the task. While recent saliency models manage to predict the users’ visual attention in visualizations during exploratory analysis, there is little evidence how much influence bottom-up saliency has on task-based visual analysis. Therefore, we performed an eye-tracking study with 47 users to determine the users’ path of attention when solving three low-level analytical tasks using 30 different charts from the MASSVIS database [1]. We also compared our task-based eye tracking data to the data from the original memorability experiment by Borkin et al. [2]. We found that solving a task leads to more consistent viewing patterns compared to exploratory visual analysis. However, bottom-up saliency of a visualization has negligible influence on users’ fixations and task efficiency when performing a low-level analytical task. Also, the efficiency of visual search for an extreme target data point is barely influenced by the target’s bottom-up saliency. Therefore, we conclude that bottom-up saliency models tailored towards information visualization are not suitable for predicting visual attention when performing task-based visual analysis. We discuss potential reasons and suggest extensions to visual attention models to better account for task-based visual analysis.

[1]  Jonathan Tran,et al.  Using Eye Tracking Metrics and Visual Saliency Maps to Assess Image Utility , 2016, HVEI.

[2]  Michelle A. Borkin,et al.  Eye Fixation Metrics for Large Scale Evaluation and Comparison of Information Visualizations , 2015, ETVIS.

[3]  Min Chen,et al.  Eurographics/ Ieee-vgtc Symposium on Visualization 2010 a Salience-based Quality Metric for Visualization , 2022 .

[4]  Qi Zhao,et al.  SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  Laurent Itti,et al.  Realistic avatar eye and head animation using a neurobiological model of visual attention , 2004, SPIE Optics + Photonics.

[6]  Simone Garlandini,et al.  Evaluating the Effectiveness and Efficiency of Visual Variables for Geographic Information Visualization , 2009, COSIT.

[7]  John K. Tsotsos,et al.  Saliency Based on Information Maximization , 2005, NIPS.

[8]  Martin A. Giese,et al.  Hooligan detection: The effects of saliency and expert knowledge , 2011, ICDP.

[9]  Frédo Durand,et al.  Learning Visual Importance for Graphic Designs and Data Visualizations , 2017, UIST.

[10]  Joseph H. Goldberg,et al.  Eye tracking for visualization evaluation: Reading values on linear versus radial graphs , 2011, Inf. Vis..

[11]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[12]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[13]  James R. Eagan,et al.  Low-level components of analytic activity in information visualization , 2005, IEEE Symposium on Information Visualization, 2005. INFOVIS 2005..

[14]  Michelle A. Borkin,et al.  What Makes a Visualization Memorable? , 2013, IEEE Transactions on Visualization and Computer Graphics.

[15]  Stephan Diehl,et al.  Comparing the Readability of Graph Layouts using Eyetracking and Task-oriented Analysis , 2009, CAe.

[16]  Eduard Gröller,et al.  Exploring visual prominence of multi-channel highlighting in visualizations , 2017, SCCG.

[17]  Stan Sclaroff,et al.  Saliency Detection: A Boolean Map Approach , 2013, 2013 IEEE International Conference on Computer Vision.

[18]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[19]  David Whitney,et al.  How Capacity Limits of Attention Influence Information Visualization Effectiveness , 2012, IEEE Transactions on Visualization and Computer Graphics.

[20]  Liaoyuan Zeng,et al.  Saliency detection in complex scenes , 2014, EURASIP J. Image Video Process..

[21]  Ali Borji,et al.  CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research , 2015, ArXiv.

[22]  Harri Siirtola,et al.  Visual Perception of Parallel Coordinate Visualizations , 2009, 2009 13th International Conference Information Visualisation.

[23]  Michael Burch,et al.  Evaluation of Traditional, Orthogonal, and Radial Tree Diagrams by an Eye Tracking Study , 2011, IEEE Transactions on Visualization and Computer Graphics.

[24]  Tim K Marks,et al.  SUN: A Bayesian framework for saliency using natural statistics. , 2008, Journal of vision.

[25]  Amitabh Varshney,et al.  Saliency-guided Enhancement for Volume Visualization , 2006, IEEE Transactions on Visualization and Computer Graphics.

[26]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[27]  Joseph H. Goldberg,et al.  Comparing information graphics: a critical look at eye tracking , 2010, BELIV '10.

[28]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[29]  Xiaolin Hu,et al.  Predicting Eye Fixations With Higher-Level Visual Features , 2015, IEEE Transactions on Image Processing.

[30]  Hsueh-Cheng Wang,et al.  The attraction of visual attention to texts in real-world scenes. , 2012, Journal of vision.

[31]  Rita Cucchiara,et al.  Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model , 2016, IEEE Transactions on Image Processing.

[32]  Hanspeter Pfister,et al.  Beyond Memorability: Visualization Recognition and Recall , 2016, IEEE Transactions on Visualization and Computer Graphics.

[33]  Garrison W. Cottrell,et al.  Visual saliency model for robot cameras , 2008, 2008 IEEE International Conference on Robotics and Automation.

[34]  Weisi Lin,et al.  Selective rendering with graphical saliency model , 2011, 2011 IEEE 10th IVMSP Workshop: Perception and Visual Signal Analysis.

[35]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[36]  Michael Burch,et al.  Evaluating visual analytics with eye tracking , 2014, BELIV.

[37]  Sung-Hee Kim,et al.  Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making , 2012, IEEE Transactions on Visualization and Computer Graphics.

[38]  Stan Sclaroff,et al.  Exploiting Surroundedness for Saliency Detection: A Boolean Map Approach , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Lihi Zelnik-Manor,et al.  Context-Aware Saliency Detection , 2012, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Gabriela Csurka,et al.  A framework for visual saliency detection with applications to image thumbnailing , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[42]  Michael J. Haass,et al.  Data Visualization Saliency Model: A Tool for Evaluating Abstract Data Visualizations , 2018, IEEE Transactions on Visualization and Computer Graphics.

[43]  Christof Koch,et al.  Image Signature: Highlighting Sparse Salient Regions , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  George Adelman,et al.  Encyclopedia of neuroscience , 2004 .

[45]  Sabine Süsstrunk,et al.  Saliency detection for content-aware image resizing , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[46]  Laurent Itti,et al.  Biologically Inspired Mobile Robot Vision Localization , 2009, IEEE Transactions on Robotics.

[47]  Frédo Durand,et al.  What Do Different Evaluation Metrics Tell Us About Saliency Models? , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Gert Kootstra,et al.  Paying Attention to Symmetry , 2008, BMVC.

[49]  Michael Dorr,et al.  Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Thomas Ertl,et al.  Visual analysis of perceptual and cognitive processes , 2014, 2014 International Conference on Information Visualization Theory and Applications (IVAPP).

[51]  Michael Burch,et al.  Visual task solution strategies in tree diagrams , 2013, 2013 IEEE Pacific Visualization Symposium (PacificVis).

[52]  Hans-Peter Seidel,et al.  Perceptually-Driven Visibility Optimization for Categorical Data Visualization. , 2012, IEEE transactions on visualization and computer graphics.

[53]  Asha Iyer,et al.  Components of bottom-up gaze allocation in natural images , 2005, Vision Research.

[54]  Ali Borji,et al.  Analysis of Scores, Datasets, and Models in Visual Saliency Prediction , 2013, 2013 IEEE International Conference on Computer Vision.

[55]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[56]  John K. Tsotsos,et al.  Saliency, attention, and visual search: an information theoretic approach. , 2009, Journal of vision.