Combining user logging with eye tracking for interactive and dynamic applications

User evaluations of interactive and dynamic applications face various challenges related to the active nature of these displays. For example, users can often zoom and pan on digital products, and these interactions cause changes in the extent and/or level of detail of the stimulus. Therefore, in eye tracking studies, when a user’s gaze is at a particular screen position (gaze position) over a period of time, the information contained in this particular position may have changed. Such digital activities are commonplace in modern life, yet it has been difficult to automatically compare the changing information at the viewed position, especially across many participants. Existing solutions typically involve tedious and time-consuming manual work. In this article, we propose a methodology that can overcome this problem. By combining eye tracking with user logging (mouse and keyboard actions) with cartographic products, we are able to accurately reference screen coordinates to geographic coordinates. This referencing approach allows researchers to know which geographic object (location or attribute) corresponds to the gaze coordinates at all times. We tested the proposed approach through two case studies, and discuss the advantages and disadvantages of the applied methodology. Furthermore, the applicability of the proposed approach is discussed with respect to other fields of research that use eye tracking—namely, marketing, sports and movement sciences, and experimental psychology. From these case studies and discussions, we conclude that combining eye tracking and user-logging data is an essential step forward in efficiently studying user behavior with interactive and static stimuli in multiple research fields.

[1]  Marc Antrop,et al.  Eye-tracking Analysis in Landscape Perception Research: Influence of Photograph Properties and Landscape Characteristics , 2014 .

[2]  Richard Atterer Logging Usage of AJAX Applications With the “ UsaProxy ” HTTP Proxy , 2006 .

[3]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[4]  Joseph H. Goldberg,et al.  Eye tracking in web search tasks: design implications , 2002, ETRA.

[5]  Victoria Johansson,et al.  Combined eyetracking and keystroke-logging methods for studying cognitive processes in text production , 2009, Behavior research methods.

[6]  Robert J. K. Jacob,et al.  Eye tracking in human-computer interaction and usability research : Ready to deliver the promises , 2002 .

[7]  Mark Harrower,et al.  Designing Better Map Interfaces: A Framework for Panning and Zooming , 2005, Trans. GIS.

[8]  Luca Cian,et al.  This Logo Moves Me: Dynamic Imagery from Static Images , 2014 .

[9]  Annemiek Drunen,et al.  Exploring workload and attention measurements with uLog mouse data , 2009, Behavior research methods.

[10]  Nancy Millette,et al.  How People Look at Pictures , 1935 .

[11]  Tommy Strandvall,et al.  Eye Tracking in Human-Computer Interaction and Usability Research , 2009, INTERACT.

[12]  Paul D. Allopenna,et al.  Tracking the Time Course of Spoken Word Recognition Using Eye Movements: Evidence for Continuous Mapping Models , 1998 .

[13]  Markus Huff,et al.  DynAOI: A tool for matching eye-movement data with dynamic areas of interest in animations and movies , 2010, Behavior research methods.

[14]  Keeping an eye on recruiter behavior , .

[15]  Ronald A. Rensink Change detection. , 2002, Annual review of psychology.

[16]  Kirsi Virrantaus,et al.  ICA Research Agenda on Cartography and GIScience , 2009 .

[17]  C.P.J.M. van Elzakker,et al.  Neogeography Map Users and Uses , 2012 .

[18]  Michael P. Peterson Evaluating Mapping APIs , 2014, CARTOCON.

[19]  Erik D. Reichle,et al.  Using E-Z reader to model the effects of higher level language processing on eye movements during reading , 2009, Psychonomic bulletin & review.

[20]  Simone Garlandini,et al.  Evaluating the Effectiveness and Efficiency of Visual Variables for Geographic Information Visualization , 2009, COSIT.

[21]  Frank Witlox,et al.  Investigating the Effectiveness of an Efficient Label Placement Method Using Eye Movement Data , 2012 .

[22]  Bryan Reimer,et al.  Detecting eye movements in dynamic environments , 2006, Behavior research methods.

[23]  Mario J. López,et al.  Instructional effectiveness of a computer-supported program for teaching reading comprehension strategies , 2012, Comput. Educ..

[24]  G. Underwood,et al.  Eye movements and hazard perception in police pursuit and emergency response driving. , 2003, Journal of experimental psychology. Applied.

[25]  Weihua Dong,et al.  Eye Tracking to Explore the Potential of Enhanced Imagery Basemaps in Web Mapping , 2014 .

[26]  Menno-Jan Kraak,et al.  Web cartography : developments and prospects , 2001 .

[27]  Arzu Çöltekin,et al.  Eye-Hand Coordination During Visual Search on Geographic Displays , 2014, ET4S@GIScience.

[28]  Robert W. Reeder,et al.  WebEyeMapper and WebLogger: tools for analyzing eye tracking data collected in web-use studies , 2001, CHI Extended Abstracts.

[29]  Lars Brodersen,et al.  Applying eye-movement tracking for the study of map perception and map design , 2001 .

[30]  R. Pieters,et al.  A Review of Eye-Tracking Research in Marketing , 2008 .

[31]  Philippe De Maeyer,et al.  Comparing Paper and Digital Topographic Maps Using Eye Tracking , 2014, CARTOCON.

[32]  Michel Wedel,et al.  Eye tracking for visual marketing , 2008 .

[33]  Erik Hazzard,et al.  OpenLayers 2.10 Beginner's Guide , 2011 .

[34]  Arzu Çöltekin,et al.  Exploring the efficiency of users' visual analytics strategies based on sequence analysis of eye movement recordings , 2010, Int. J. Geogr. Inf. Sci..

[35]  B. Schneiderman,et al.  Designing the User Interface. Strategies for Effective Human-Computer Interaction , 1992 .

[36]  Eric T. Bradlow,et al.  Does In-Store Marketing Work? Effects of the Number and Position of Shelf Facings on Brand Attention and Evaluation at the Point of Purchase , 2009 .

[37]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[38]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[39]  Sara Irina Fabrikant,et al.  Introduction: Cognitive Issues in Geographic Information Visualization , 2009, Cartogr. Int. J. Geogr. Inf. Geovisualization.

[40]  Menno-Jan Kraak,et al.  A Methodological Framework for Researching the Usability of the Space-Time Cube , 2013 .

[41]  M. A. Recarte,et al.  Effects of verbal and spatial-imagery tasks on eye fixations while driving. , 2000 .

[42]  Jan Brus,et al.  Modern Trends in Cartography: Selected Papers of CARTOCON 2014 , 2015, CARTOCON.

[43]  K. Rayner The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search , 2009, Quarterly journal of experimental psychology.

[44]  Luc Crevits,et al.  Saccadic eye movements and finger reaction times of table tennis players of different levels , 2000 .

[45]  David F. Redmiles,et al.  Extracting usability information from user interface events , 2000, CSUR.

[46]  Linden J. Ball,et al.  Eye tracking in HCI and usability research. , 2006 .

[47]  Arzu Çöltekin,et al.  Evaluating the Effectiveness of Interactive Map Interface Designs: A Case Study Integrating Usability Metrics with Eye-Movement Analysis , 2009 .

[48]  Ben Shneiderman,et al.  Designing the user interface - strategies for effective human-computer interaction, 2nd Edition , 1992 .

[49]  Greet Cardon,et al.  A hazard perception test for cycling children: an exploratory study , 2013, ETSA '13.

[50]  Rik Pieters,et al.  Attention Capture and Transfer in Advertising: Brand, Pictorial, and Text-Size Effects , 2004 .

[51]  B. Kahn,et al.  The "visual preference heuristic": The influence of visual versus verbal depiction on assortment processing, perceived variety, and choice overload , 2014 .

[52]  D. R. Montello Cognitive Research in GIScience: Recent Achievements and Future Prospects , 2009 .

[53]  Claude Ghaoui,et al.  Encyclopedia of Human Computer Interaction , 2005 .

[54]  Xia Li,et al.  Visual Exploration of Eye Movement Data Using the Space-Time-Cube , 2010, GIScience.

[55]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[56]  Mary Hegarty,et al.  Thinking about the weather: How display salience and knowledge affect performance in a graphic inference task. , 2010, Journal of experimental psychology. Learning, memory, and cognition.

[57]  Thierry Poibeau,et al.  Introduction: Cognitive Issues in Natural Language Processing , 2016, TAL.

[58]  Veerle Van Eetvelde,et al.  Analyzing the perception of water surfaces in urban landscapes using eye tracking , 2013 .

[59]  R. Hartsuiker,et al.  The influence of semantic constraints on bilingual word recognition during sentence reading , 2011 .

[60]  K. Rayner Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[61]  Laura Chamberlain Eye Tracking Methodology; Theory and Practice , 2007 .

[62]  Paul M. Fitts,et al.  Eye movements of aircraft pilots during instrument-landing approaches. , 1950 .

[63]  R. Vaeyens,et al.  CUE USAGE IN VOLLEYBALL: A TIME COURSE COMPARISON OF ELITE, INTERMEDIATE AND NOVICE FEMALE PLAYERS , 2014, Biology of sport.

[64]  Fabio Paternò,et al.  Intelligent analysis of user interactions with web applications , 2002, IUI '02.

[65]  Greet Cardon,et al.  The visual control of bicycle steering: The effects of speed and path width. , 2013, Accident; analysis and prevention.

[66]  Stephan Winter,et al.  Spatial Information Theory, 8th International Conference, COSIT 2007, Melbourne, Australia, September 19-23, 2007, Proceedings , 2007, COSIT.

[67]  Michael P. Peterson Maps and the internet. , 2003 .

[68]  Alex Singleton,et al.  Web mapping 2.0: The neogeography of the GeoWeb , 2008 .

[69]  A. Williams,et al.  Mechanisms Underpinning Successful Decision Making in Skilled Youth Soccer Players: An Analysis of Visual Search Behaviors , 2007, Journal of motor behavior.

[70]  Linden J. Ball,et al.  Eye Tracking in Human-Computer Interaction and Usability Research : Current Status and Future Prospects , 2004 .

[71]  Henry W. Castner,et al.  Eye-Movement Parameters and Perceived Map Complexity—II , 1984 .

[72]  M. A. Recarte,et al.  Effects of verbal and spatial-imagery tasks on eye fixations while driving. , 2000, Journal of experimental psychology. Applied.

[73]  John P. Snyder,et al.  Map Projections: A Working Manual , 2012 .

[74]  Robert E. Roth,et al.  Extending the Google Maps API for Event Animation Mashups , 2009 .

[75]  Ann Blandford,et al.  Human factors in computing , 2003 .

[76]  A. Pellicer‐Sánchez INCIDENTAL L2 VOCABULARY ACQUISITION FROM AND WHILE READING , 2015, Studies in Second Language Acquisition.

[77]  Albrecht Schmidt,et al.  Knowing the User's Every Move – User Activity Tracking for Website Usability Evaluation and Implicit Interaction , 2006 .

[78]  Michael W. Dobson,et al.  Eye Movement Parameters and Map Reading , 1977 .

[79]  Alzbeta Brychtova,et al.  Eye-tracking Study on Different Perception of 2D and 3D Terrain Visualisation , 2013 .

[80]  Heiner Deubel,et al.  The mind's eye : cognitive and applied aspects of eye movement research , 2003 .

[81]  Theodore Steinke Eye Movement Studies In Cartography And Related Fields , 1987 .

[82]  Eyad Elyan,et al.  The cognitive benefits of dynamic representations in the acquisition of spatial navigation skills , 2014, Comput. Hum. Behav..

[83]  A. MacEachren,et al.  Research Challenges in Geovisualization , 2001, KN - Journal of Cartography and Geographic Information.

[84]  Marc Brysbaert,et al.  Interhemispheric Communication Influences Reading Behavior , 2013, Journal of Cognitive Neuroscience.

[85]  Mark Harrower,et al.  The Cognitive Limits of Animated Maps , 2007, Cartogr. Int. J. Geogr. Inf. Geovisualization.

[86]  C.P.J.M. van Elzakker,et al.  Focus on geoinformation users : cognitive and use / user issues in contemporary cartography , 2013 .

[87]  William Cartwright Neocartography: Opportunities, Issues and Prospects , 2012 .

[88]  Wai-Tat Fu,et al.  A user-tracing architecture for modeling interaction with the world wide web , 2002, AVI '02.

[89]  Michael S. Ambinder,et al.  Change blindness , 1997, Trends in Cognitive Sciences.

[90]  Martin Raubal,et al.  Where Am I? Investigating Map Matching During Self‐Localization With Mobile Eye Tracking in an Urban Environment , 2014, Trans. GIS.

[91]  Alan M. MacEachren,et al.  How Maps Work - Representation, Visualization, and Design , 1995 .

[92]  Tiina. Sarjakoski USABILITY EVALUATION OF TOPOGRAPHIC MAPS IN MOBILE DEVICES , 2003 .

[93]  Bin Jiang,et al.  Cognitive and Usability Issues in Geovisualization , 2001 .

[94]  Sara Irina Fabrikant,et al.  How users interact with a 3D geo-browser under time pressure , 2013 .

[95]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[96]  Candice R. Luebbering,et al.  Expanding Display Size and Resolution for Viewing Geospatial Data: A User Study with Multiple-Monitor High-Resolution Displays , 2008 .

[97]  Menno-Jan Kraak,et al.  Conflicts in Neogeography Maps , 2012 .

[98]  Kerry Rodden,et al.  Exploring How Mouse Movements Relate to Eye Movements on Web Search Results Pages , 2007 .

[99]  Gennady Andrienko,et al.  Novel Method to Measure Inference Affordance in Static Small-Multiple Map Displays Representing Dynamic Processes , 2008 .

[100]  Arzu Çöltekin,et al.  Understanding Soil Acidification Process Using Animation and Text: An Empirical User Evaluation With Eye Tracking , 2014 .

[101]  Stephen A. Brewster,et al.  Overcoming the Lack of Screen Space on Mobile Computers , 2002, Personal and Ubiquitous Computing.

[102]  Robert E. Roth,et al.  Interacting with maps: The science and practice of cartographic interaction , 2011 .