Understanding Visual Saliency in Mobile User Interfaces

For graphical user interface (UI) design, it is important to understand what attracts visual attention. While previous work on saliency has focused on desktop and web-based UIs, mobile app UIs differ from these in several respects. We present findings from a controlled study with 30 participants and 193 mobile UIs. The results speak to a role of expectations in guiding where users look at. Strong bias toward the top-left corner of the display, text, and images was evident, while bottom-up features such as color or size affected saliency less. Classic, parameter-free saliency models showed a weak fit with the data, and data-driven models improved significantly when trained specifically on this dataset (e.g., NSS rose from 0.66 to 0.84). We also release the first annotated dataset for investigating visual saliency in mobile UIs.

[1]  Laurent Itti,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Rapid Biologically-inspired Scene Classification Using Features Shared with Visual Attention , 2022 .

[2]  Sergey Bezryadin,et al.  Brightness Calculation in Digital Image Processing , 2007 .

[3]  Shijian Lu,et al.  Saliency Modeling from Image Histograms , 2012, ECCV.

[4]  F. Pellandini,et al.  Adaptive color image compression based on visual attention , 2001, Proceedings 11th International Conference on Image Analysis and Processing.

[5]  Stan Sclaroff,et al.  Saliency Detection: A Boolean Map Approach , 2013, 2013 IEEE International Conference on Computer Vision.

[6]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[7]  Qi Zhao,et al.  Webpage Saliency , 2014, ECCV.

[8]  Samit Bhattacharya,et al.  Quantitative visual attention prediction on webpage images using multiclass SVM , 2019, ETRA.

[9]  IttiLaurent,et al.  Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention , 2007 .

[10]  Ruth Rosenholtz,et al.  Do predictions of visual perception aid design? , 2011, TAP.

[11]  G. Underwood,et al.  The potency of people in pictures: evidence from sequences of eye fixations. , 2010, Journal of vision.

[12]  Jorma Laaksonen,et al.  Saliency Revisited: Analysis of Mouse Movements Versus Fixations , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Bingbing Ni,et al.  Touch saliency , 2012, ACM Multimedia.

[14]  L. Itti,et al.  Quantifying center bias of observers in free viewing of dynamic natural scenes. , 2009, Journal of vision.

[15]  Laurent Itti,et al.  Visual salience , 2007, Scholarpedia.

[16]  Loong Fah Cheong,et al.  Active segmentation with fixation , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[17]  Julie E. Boland,et al.  Cultural variation in eye movements during scene perception. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  P. König,et al.  Spatial biases in viewing behavior. , 2014, Journal of vision.

[20]  Hsueh-Cheng Wang,et al.  The attraction of visual attention to texts in real-world scenes. , 2012, Journal of vision.

[21]  Rita Cucchiara,et al.  Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model , 2016, IEEE Transactions on Image Processing.

[22]  Henrik I. Christensen,et al.  Computational visual attention systems and their cognitive foundations: A survey , 2010, TAP.

[23]  Nathalie Guyader,et al.  Improving Visual Saliency by Adding ‘Face Feature Map’ and ‘Center Bias’ , 2012, Cognitive Computation.

[24]  Bingbing Ni,et al.  Touch Saliency: Characteristics and Prediction , 2014, IEEE Transactions on Multimedia.

[25]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[26]  Ali Borji,et al.  Analysis of Scores, Datasets, and Models in Visual Saliency Prediction , 2013, 2013 IEEE International Conference on Computer Vision.

[27]  Derrick J. Parkhurst,et al.  Modeling the role of salience in the allocation of overt visual attention , 2002, Vision Research.

[28]  Thomas F. Liu,et al.  Learning Design Semantics for Mobile Apps , 2018, UIST.

[29]  Jennifer Long,et al.  Viewing distance and eyestrain symptoms with prolonged viewing of smartphones , 2017, Clinical & experimental optometry.

[30]  Aaron R. Seitz,et al.  Learning what to expect (in visual perception) , 2013, Front. Hum. Neurosci..

[31]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  L. Cooke,et al.  Is the Mouse a “ Poor Man ’ s Eye Tracker ” ? , 2006 .

[33]  Frédo Durand,et al.  A Benchmark of Computational Models of Saliency to Predict Human Fixations , 2012 .

[34]  J. Henderson.  Eye movement control during visual object processing: effects of initial fixation position and semantic constraint. , 1993, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[35]  Frédo Durand,et al.  What Do Different Evaluation Metrics Tell Us About Saliency Models? , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[37]  Ali Borji,et al.  Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges , 2018, 1810.03716.

[38]  Qi Zhao,et al.  Learning saliency-based visual attention: A review , 2013, Signal Process..

[39]  Hae-Young Kim,et al.  Statistical notes for clinical researchers: Chi-squared test and Fisher's exact test , 2017, Restorative dentistry & endodontics.

[40]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Stephen J. Payne,et al.  Skim reading by satisficing: evidence from eye tracking , 2011, CHI.

[42]  Elena A. Fedorovskaya,et al.  On the Role of Color in Visual Saliency , 2017, HVEI.

[43]  Pingmei Xu,et al.  Spatio-Temporal Modeling and Prediction of Visual Attention in Graphical User Interfaces , 2016, CHI.

[44]  John K. Tsotsos.  Is complexity theory appropriate for analyzing biological systems? , 1991, Behavioral and Brain Sciences.

[45]  Benjamin W Tatler,et al.  The central fixation bias in scene viewing: selecting an optimal viewing position independently of motor biases and image feature distributions. , 2007, Journal of vision.

[46]  J. Wolfe,et al.  What attributes guide the deployment of visual attention and how do they do it? , 2004, Nature Reviews Neuroscience.

[47]  Antti Oulasvirta,et al.  Adaptive feature guidance: Modelling visual search with graphical layouts , 2020, Int. J. Hum. Comput. Stud..

[48]  Gitte Lindgaard,et al.  Attention web designers: You have 50 milliseconds to make a good first impression! , 2006, Behav. Inf. Technol..

[49]  Simon P Liversedge,et al.  Rayner's 1979 Paper , 2009 .

[50]  Lie Lu,et al.  A generic framework of user attention model and its application in video summarization , 2005, IEEE Trans. Multim..

[51]  C. Koch,et al.  Faces and text attract gaze independent of the task: Experimental data and computer model. , 2009, Journal of vision.

[52]  Antonella De Angeli,et al.  Visual impressions of mobile app interfaces , 2014, NordiCHI.

[53]  Qi Zhao,et al.  SALICON: Saliency in Context , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Frédo Durand,et al.  Learning Visual Importance for Graphic Designs and Data Visualizations , 2017, UIST.

[55]  Ali Borji,et al.  Understanding and Visualizing Deep Visual Saliency Models , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Ziad M Hafed,et al.  How is visual salience computed in the brain? Insights from behaviour, neurobiology and modelling , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[57]  Sourav Pal,et al.  Saliency Prediction for Mobile User Interfaces , 2017, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[58]  J. Henderson,et al.  Object-based attentional selection in scene viewing. , 2010, Journal of vision.

[59]  R. Caldara,et al.  Culture Shapes Eye Movements for Visually Homogeneous Objects , 2010, Front. Psychology.

[60]  Yizhou Yu,et al.  Visual saliency based on multiscale deep features , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[61]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[62]  Christopher M. Masciocchi,et al.  A Saliency Model Predicts Fixations in Web Interfaces , 2010 .

[63]  Nathalie Guyader,et al.  Contribution of Color Information in Visual Saliency Model for Videos , 2014, ICISP.