Visualization of Eye Tracking Data: A Taxonomy and Survey

This survey provides an introduction into eye tracking visualization with an overview of existing techniques. Eye tracking is important for evaluating user behaviour. Analysing eye tracking data is typically done quantitatively, applying statistical methods. However, in recent years, researchers have been increasingly using qualitative and exploratory analysis methods based on visualization techniques. For this state‐of‐the‐art report, we investigated about 110 research papers presenting visualization techniques for eye tracking data. We classified these visualization techniques and identified two main categories: point‐based methods and methods based on areas of interest. Additionally, we conducted an expert review asking leading eye tracking experts how they apply visualization techniques in their analysis of eye tracking data. Based on the experts' feedback, we identified challenges that have to be tackled in the future so that visualizations will become even more widely applied in eye tracking research.

[1]  Ling Xia,et al.  Eye tracking and online search: Lessons learned and challenges ahead , 2008, J. Assoc. Inf. Sci. Technol..

[2]  Nadir Weibel,et al.  Let's look at the cockpit: exploring mobile eye-tracking for observational research on the flight deck , 2012, ETRA.

[3]  Hansjörg Schmauder,et al.  Saccade plots , 2014, ETRA.

[4]  Nelson Alves,et al.  Combining Spatial and Temporal Information of Eye Movements in Goal-Oriented Tasks , 2013, SouthCHI.

[5]  Harri Siirtola,et al.  Visual Perception of Parallel Coordinate Visualizations , 2009, 2009 13th International Conference Information Visualisation.

[6]  Lucy T. Nowell,et al.  ThemeRiver: Visualizing Thematic Changes in Large Document Collections , 2002, IEEE Trans. Vis. Comput. Graph..

[7]  T. Smith,et al.  Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes. , 2013, Journal of vision.

[8]  Ed H. Chi,et al.  A taxonomy of visualization techniques using the data state reference model , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[9]  TsangHoi Ying,et al.  eSeeTrack—Visualizing Sequential Fixation Patterns , 2010 .

[10]  Joseph H. Goldberg,et al.  Visual scanpath representation , 2010, ETRA.

[11]  Thomas Ertl,et al.  Triangulating user behavior using eye movement, interaction, and think aloud data , 2016, ETRA.

[12]  Joseph H. Goldberg,et al.  Scanpath clustering and aggregation , 2010, ETRA.

[13]  Thies Pfeiffer Measuring and visualizing attention in space with 3D attention volumes , 2012, ETRA '12.

[14]  Moreno I. Coco The statistical challenge of scan-path analysis , 2009, 2009 2nd Conference on Human System Interactions.

[15]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[16]  Helen C. Purchase,et al.  Which Aesthetic has the Greatest Effect on Human Understanding? , 1997, GD.

[17]  Alan Kingstone,et al.  Recurrence quantification analysis of eye movements , 2013, Behavior Research Methods.

[18]  Daniel Weiskopf,et al.  AOI Rivers for Visualizing Dynamic Eye Gaze Frequencies , 2013, Comput. Graph. Forum.

[19]  R. Karsh,et al.  Cognitive strategies for visual search. , 1986, Acta psychologica.

[20]  M. Stella Atkins,et al.  Saccadic delays on targets while watching videos , 2012, ETRA.

[21]  Xia Li,et al.  Visual Exploration of Eye Movement Data Using the Space-Time-Cube , 2010, GIScience.

[22]  Melanie Tory,et al.  eSeeTrack—Visualizing Sequential Fixation Patterns , 2010, IEEE Transactions on Visualization and Computer Graphics.

[23]  Thies Pfeiffer,et al.  EyeSee3D 2.0: model-based real-time analysis of mobile eye-tracking in static and dynamic three-dimensional scenes , 2016, ETRA.

[24]  Chris Lankford,et al.  Gazetracker: software designed to facilitate eye movement analysis , 2000, ETRA.

[25]  Yuanzhen Li,et al.  Feature congestion: a measure of display clutter , 2005, CHI.

[26]  Jana Holsanova,et al.  Picture Viewing and Picture Description: Two Windows on the Mind. , 2001 .

[27]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[28]  Tamara Munzner,et al.  A Multi-Level Typology of Abstract Visualization Tasks , 2013, IEEE Transactions on Visualization and Computer Graphics.

[29]  S. G. Nikolov,et al.  Volume visualisation via region enhancement around an observer's fixation point , 2000 .

[30]  C. Latimer,et al.  Eye-movement data: Cumulative fixation time and cluster analysis , 1988 .

[31]  Agnieszka Szarkowska,et al.  The Effects of Shot Changes on Eye Movements in Subtitling , 2013 .

[32]  Lars Linsen,et al.  EyeC: Coordinated Views for Interactive Visual Exploration of Eye-Tracking Data , 2013, 2013 17th International Conference on Information Visualisation.

[33]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[34]  Robert A. Marino,et al.  Free viewing of dynamic stimuli by humans and monkeys. , 2009, Journal of vision.

[35]  Thomas Ertl,et al.  VA2: A Visual Analytics Approach for Evaluating Visual Analytics Applications , 2016, IEEE Transactions on Visualization and Computer Graphics.

[36]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[37]  Kenji Itoh,et al.  Eye-Movement Analysis of Track Monitoring Patterns of Night Train Operators: Effects of Geographic Knowledge and Fatigue , 2000 .

[38]  Paul M. Fitts,et al.  Eye movements of aircraft pilots during instrument-landing approaches. , 1950 .

[39]  John Paulin Hansen,et al.  New technological windows into mind: there is more in eyes and brains for human-computer interaction , 1996, CHI.

[40]  Thomas Ertl,et al.  AOI hierarchies for visual exploration of fixation sequences , 2016, ETRA.

[41]  L. Stark,et al.  Scanpaths in saccadic eye movements while viewing and recognizing patterns. , 1971, Vision research.

[42]  D. S. Wooding,et al.  Fixation maps: quantifying eye-movement traces , 2002, ETRA.

[43]  David Beymer,et al.  WebGazeAnalyzer: a system for capturing and analyzing web reading behavior using eye gaze , 2005, CHI Extended Abstracts.

[44]  Raimund Dachselt,et al.  Trends and Techniques in Visual Gaze Analysis , 2010, ArXiv.

[45]  L. Young,et al.  Survey of eye movement recording methods , 1975 .

[46]  Veranika Lim,et al.  Tracking Eyes in Service Prototyping , 2013, INTERACT.

[47]  Johann Schrammel,et al.  3D attention: measurement of visual saliency using eye tracking glasses , 2013, CHI Extended Abstracts.

[48]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[49]  Tamara Munzner,et al.  Process and Pitfalls in Writing Information Visualization Research Papers , 2008, Information Visualization.

[50]  Pieter J. Blignaut,et al.  Visual span and other parameters for the generation of heatmaps , 2010, ETRA.

[51]  J. V. van Berkum,et al.  How robust is the language architecture? The case of mood , 2013, Front. Psychol..

[52]  John Paulin Hansen,et al.  COGNITIVE MODELLING OF A SHIP NAVIGATOR BASED ON PROTOCOL AND EYE-MOVEMENT ANALYSIS , 1998 .

[53]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[54]  Daniel C. Richardson,et al.  Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension , 2005, Cogn. Sci..

[55]  Arjan Kuijper,et al.  Visual Analysis of Large Graphs: State‐of‐the‐Art and Future Research Challenges , 2011, Eurographics.

[56]  M. Sheelagh T. Carpendale,et al.  A Review of Temporal Data Visualizations Based on Space-Time Cube Operations , 2014, EuroVis.

[57]  Eli Peli,et al.  Where people look when watching movies: Do all viewers look at the same place? , 2007, Comput. Biol. Medicine.

[58]  Heiner Deubel,et al.  The mind's eye : cognitive and applied aspects of eye movement research , 2003 .

[59]  T. Hägerstrand DIORAMA, PATH AND PROJECT , 1982 .

[60]  Jean-Daniel Fekete,et al.  Task taxonomy for graph visualization , 2006, BELIV '06.

[61]  Sung-Hee Kim,et al.  Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making , 2012, IEEE Transactions on Visualization and Computer Graphics.

[62]  Cristina Conati,et al.  Individual user characteristics and information visualization: connecting the dots through eye tracking , 2013, CHI.

[63]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[64]  Byron Nakos,et al.  EyeMMV toolbox: An eye movement post-analysis tool based on a two-step spatial dispersion threshold for fixation identification , 2014 .

[65]  Ignace Hooge,et al.  Scan path entropy and arrow plots: capturing scanning behavior of multiple observers , 2013, Front. Psychol..

[66]  Kowa Koida,et al.  Color vision test for dichromatic and trichromatic macaque monkeys. , 2013, Journal of vision.

[67]  Jeff B. Pelz,et al.  Learning eye movement patterns for characterization of perceptual expertise , 2012, ETRA.

[68]  Jinwook Seo,et al.  GazeVis: Interactive 3D Gaze Visualization for Contiguous Cross-Sectional Medical Images , 2014, IEEE Transactions on Visualization and Computer Graphics.

[69]  Björn Peters,et al.  Hearing loss and a supportive tactile signal in a navigation system : effects on driving behavior and eye movements , 2013 .

[70]  C. Trepagnier,et al.  Gaze data visualization tools: opportunities and challenges , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[71]  Thies Pfeiffer,et al.  EyeSee3D: a low-cost approach for analyzing mobile 3D eye tracking data using computer vision and augmented reality technology , 2014, ETRA.

[72]  Thomas Ertl,et al.  Application design for an eye tracking analysis based on visual analytics , 2016, ETRA.

[73]  Markus Huff,et al.  DynAOI: A tool for matching eye-movement data with dynamic areas of interest in animations and movies , 2010, Behavior research methods.

[74]  Minoru Nakayama,et al.  Estimation of viewer's response for contextual understanding of tasks using features of eye-movements , 2010, ETRA '10.

[75]  Pilar Orero,et al.  Aggregate gaze visualization with real-time heatmaps , 2012, ETRA.

[76]  Jana Holsanova,et al.  Reading or scanning? A study of newspaper and net paper reading. , 2003 .

[77]  Michael Burch,et al.  State-of-the-Art of Visualization for Eye Tracking Data , 2014, EuroVis.

[78]  Michael Burch,et al.  Gaze Stripes: Image-Based Visualization of Eye Tracking Data , 2016, IEEE Transactions on Visualization and Computer Graphics.

[79]  Helge J. Ritter,et al.  Automatic analysis of 3D gaze coordinates on scene objects using data from eye-tracking and motion-capture systems , 2012, ETRA '12.

[80]  M. Just,et al.  Eye fixations and cognitive processes , 1976, Cognitive Psychology.

[81]  Tsukasa Ogasawara,et al.  Estimating 3D point-of-regard and visualizing gaze trajectories under natural head movements , 2010, ETRA '10.

[82]  Raimund Dachselt,et al.  Advanced gaze visualizations for three-dimensional virtual environments , 2010, ETRA.

[83]  Agnieszka Bojko,et al.  Informative or Misleading? Heatmaps Deconstructed , 2009, HCI.

[84]  Oleg pakov,et al.  iComponent ­ Device-Independent Platform for Analyzing Eye Movement Data and Developing Eye-Based Applications , 2008 .

[85]  N. Hari Narayanan,et al.  Comparing interfaces based on what users watch and do , 2000, ETRA.

[86]  G Schneider,et al.  Visual attention of anaesthetists during simulated critical incidents. , 2011, British journal of anaesthesia.

[87]  Jan Noyes,et al.  Scanpath Analysis of Fused Multi-Sensor Images with Luminance Change: A Pilot Study , 2006, 2006 9th International Conference on Information Fusion.

[88]  Daniel Weiskopf,et al.  AOI transition trees , 2015, Graphics Interface.

[89]  Patrick Bouthemy,et al.  Optical flow modeling and computation: A survey , 2015, Comput. Vis. Image Underst..

[90]  L. Stark,et al.  Scanpaths in Eye Movements during Pattern Perception , 1971, Science.

[91]  Sebastian Koch,et al.  Visual Analysis and Dissemination of Scientific Literature Collections with SurVis , 2016, IEEE Transactions on Visualization and Computer Graphics.

[92]  Peter Fröhlich,et al.  KIBITZER: a wearable system for eye-gaze-based mobile urban exploration , 2010, AH.

[93]  Christophe Hurter,et al.  A descriptive model of visual scanning , 2010, BELIV '10.

[94]  Andrew T Duchowski,et al.  A breadth-first survey of eye-tracking applications , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[95]  Anne R. Haake,et al.  eyePatterns: software for identifying patterns and similarities across fixation sequences , 2006, ETRA.

[96]  Bruce H. McCormick,et al.  Gaze-contingent video resolution degradation , 1998, Electronic Imaging.

[97]  Päivi Majaranta,et al.  Eye-Tracking Reveals the Personal Styles for Search Result Evaluation , 2005, INTERACT.

[98]  Sara Irina Fabrikant,et al.  How Do People View Multi-Component Animated Maps? , 2014 .

[99]  Tim Halverson,et al.  Cleaning up systematic error in eye-tracking data by using required fixation locations , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[100]  Michael Burch,et al.  Visual Analytics Methodology for Eye Movement Studies , 2012, IEEE Transactions on Visualization and Computer Graphics.

[101]  Daniel Weiskopf,et al.  Space-Time Visual Analytics of Eye-Tracking Data for Dynamic Stimuli , 2013, IEEE Transactions on Visualization and Computer Graphics.

[102]  Christophe Hurter,et al.  Bundled Visualization of DynamicGraph and Trail Data , 2014, IEEE Transactions on Visualization and Computer Graphics.

[103]  Joseph H. Goldberg,et al.  Comparing information graphics: a critical look at eye tracking , 2010, BELIV '10.

[104]  Sean P. Goggins,et al.  Assessing Multiple Perspectives in Three Dimensional Virtual Worlds: Eye Tracking and All Views Qualitative Analysis (AVQA) , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[105]  Anand K. Gramopadhye,et al.  3D eye movement analysis for VR visual inspection training , 2002, ETRA.

[106]  Jock D. Mackinlay,et al.  The structure of the information visualization design space , 1997, Proceedings of VIZ '97: Visualization Conference, Information Visualization Symposium and Parallel Rendering Symposium.

[107]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[108]  Florian Heimerl,et al.  ISeeCube: visual analysis of gaze data for video , 2014, ETRA.

[109]  Daniel Weiskopf,et al.  Eye Tracking for Personal Visual Analytics , 2015, IEEE Computer Graphics and Applications.

[110]  Andreas Dengel,et al.  Towards robust gaze-based objective quality measures for text , 2012, ETRA '12.

[111]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[112]  Menno-Jan Kraak,et al.  The space - time cube revisited from a geovisualization perspective , 2003 .

[113]  Oleg Spakov,et al.  KiEV: a tool for visualization of reading and writing processes in translation of text , 2008, ETRA '08.

[114]  Rui Rodrigues,et al.  A Television News Graphical Layout Analysis Method Using Eye Tracking , 2012, 2012 16th International Conference on Information Visualisation.

[115]  Päivi Majaranta,et al.  Static Visualization of Temporal Eye-Tracking Data , 2005, INTERACT.

[116]  Darius Miniotas,et al.  Visualization of eye gaze data using heat maps , 2007 .

[117]  Geri Gay,et al.  Averaging scan patterns and what they can tell us , 2006, ETRA.

[118]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[119]  M. Potter,et al.  Temporal constraints on conscious vision: on the ubiquitous nature of the attentional blink. , 2009, Journal of vision.

[120]  Andreas Bulling,et al.  Introduction to the PETMEI special issue , 2014 .

[121]  Lucas Paletta,et al.  A Computer Vision System for Attention Mapping in SLAM based 3D Models , 2013, ArXiv.

[122]  Andrew T. Duchowski,et al.  Group-wise similarity and classification of aggregate scanpaths , 2010, ETRA '10.

[123]  Thomas Ertl,et al.  Parallel scan-path visualization , 2012, ETRA.

[124]  John M. Henderson,et al.  Clustering of Gaze During Dynamic Scene Viewing is Predicted by Motion , 2011, Cognitive Computation.

[125]  David C Burr,et al.  Temporal auditory capture does not affect the time course of saccadic mislocalization of visual stimuli. , 2010, Journal of vision.

[126]  Michael Burch,et al.  A Task-Based View on the Visual Analysis of Eye-Tracking Data , 2015, ETVIS.

[127]  Thomas Ertl,et al.  Circular heat map transition diagram , 2013, ETSA '13.

[128]  Guang-Zhong Yang,et al.  Eyegaze analysis of displays with combined 2D and 3D views , 2005, VIS 05. IEEE Visualization, 2005..

[129]  Noriko Kando,et al.  Visualization of User Eye Movements for Search Result Pages , 2008, EVIA@NTCIR.

[130]  Aulikki Hyrskykari,et al.  101 spots, or how do users read menus? , 1998, CHI.

[131]  Lingyun Sun,et al.  Gaze-Based Annotations for Reading Comprehension , 2015, CHI.

[132]  Alfred Inselberg,et al.  The plane with parallel coordinates , 1985, The Visual Computer.

[133]  Roel Vertegaal,et al.  The GAZE groupware system: mediating joint attention in multiparty communication and collaboration , 1999, CHI '99.

[134]  Andoni Beristain,et al.  Volume Visual Attention Maps (VVAM) in Ray-Casting Rendering , 2012, MMVR.

[135]  Fabio Pellacini,et al.  Visualizing Paths in Context , 2006 .

[136]  Douglas DeCarlo,et al.  Robust clustering of eye movement recordings for quantification of visual interest , 2004, ETRA.

[137]  N H MACKWORTH,et al.  Eye fixations recorded on changing visual scenes by the television eye-marker. , 1958, Journal of the Optical Society of America.

[138]  Larry Light,et al.  Eye movements and TV viewing in children , 1964 .

[139]  Yeliz Yesilada,et al.  Eye tracking scanpath analysis techniques on web pages: A survey, evaluation and comparison , 2015 .

[140]  Marcus Nyström,et al.  Semantic override of low-level features in image viewing - both initially and overall , 2008 .

[141]  Daniel Hutcherson Shelf Real Estate : Package Spacing in a Retail Environment , .

[142]  Claudio M. Privitera,et al.  Algorithms for Defining Visual Regions-of-Interest: Comparison with Eye Fixations , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[143]  James L. Croft,et al.  Gaze behavior of spotters during an air-to-ground search , 2006, ETRA '06.

[144]  Linden J. Ball,et al.  Eye Tracking in Human-Computer Interaction and Usability Research : Current Status and Future Prospects , 2004 .

[145]  Michael Burch,et al.  A visual approach for scan path comparison , 2014, ETRA.

[146]  Raimund Dachselt,et al.  3D attentional maps: aggregated gaze visualizations in three-dimensional virtual environments , 2010, AVI.

[147]  Chris North,et al.  Exploring context switching and cognition in dual-view coordinated visualizations , 2003, Proceedings International Conference on Coordinated and Multiple Views in Exploratory Visualization - CMV 2003 -.

[148]  Erhardt Barth,et al.  Space-variant spatio-temporal filtering of video for gaze visualization and perceptual learning , 2010, ETRA.