Studying Developer Eye Movements to Measure Cognitive Workload and Visual Effort for Expertise Assessment

Eye movement data provides valuable insights that help test hypotheses about a software developer's comprehension process. The pupillary response is successfully used to assess mental processing effort and attentional focus. Relatively little is known about the impact of expertise level in cognitive effort during programming tasks. This paper presents a quantitative analysis that compares the eye movements of 207 experts and novices collected while solving program comprehension tasks. The goal is to examine changes of developers' eye movement metrics in accordance with their expertise. The results indicate significant increase in pupil size with the novice group compared to the experts, explaining higher cognitive effort for novices. Novices also tend to have a significant number of fixations and higher gaze time compared to experts when they comprehend code. Moreover, a correlation study found that programming experience is still a powerful indicator when explaining expertise in this eye-tracking dataset among other expertise variables.

[1]  Jonathan I. Maletic,et al.  Determining Differences in Reading Behavior Between Experts and Novices by Investigating Eye Movement on Source Code Constructs During a Bug Fixing Task , 2021, ETRA Short Papers.

[2]  Jonathan I. Maletic,et al.  From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study , 2021, 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC).

[3]  Westley Weimer,et al.  Relating Reading, Visualization, and Coding for New Programmers: A Neuroimaging Study , 2021, 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).

[4]  Fabian Fagerholm,et al.  EMIP: The eye movements in programming dataset , 2020, Sci. Comput. Program..

[5]  Chris Parnin,et al.  Here We Go Again: Why Is It Difficult for Developers to Learn Another Programming Language? , 2020, 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE).

[6]  Yann-Gaël Guéhéneuc,et al.  A practical guide on conducting eye tracking studies in software engineering , 2020, Empirical Software Engineering.

[7]  B. Laeng,et al.  Pupillometry , 2020, Definitions.

[8]  Christoph Treude,et al.  Comprehending Test Code: An Empirical Study , 2019, 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[9]  Jonathan I. Maletic,et al.  Using developer eye movements to externalize the mental model used in code summarization tasks , 2019, ETRA.

[10]  Andrea Kienle,et al.  The Influence of Different AOI Models in Source Code Comprehension Analysis , 2019, 2019 IEEE/ACM 6th International Workshop on Eye Movements in Programming (EMIP).

[11]  Jonathan I. Maletic,et al.  Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[12]  Jonathan I. Maletic,et al.  iTrace: eye tracking infrastructure for development environments , 2018, ETRA.

[13]  Christa Boer,et al.  Correlation Coefficients: Appropriate Use and Interpretation , 2018, Anesthesia and analgesia.

[14]  Angel Jiménez Molina,et al.  Using Psychophysiological Sensors to Assess Mental Workload During Web Browsing , 2018, Sensors.

[15]  Jonathan I. Maletic,et al.  An Empirical Study of Debugging Patterns Among Novices Programmers , 2017, SIGCSE.

[16]  Jason S. Tsukahara,et al.  The relationship between baseline pupil size and intelligence , 2016, Cognitive Psychology.

[17]  Maria K. Eckstein,et al.  Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development? , 2016, Developmental Cognitive Neuroscience.

[18]  Yann-Gaël Guéhéneuc,et al.  Eye-Tracking Metrics in Software Engineering , 2015, 2015 Asia-Pacific Software Engineering Conference (APSEC).

[19]  Yann-Gaël Guéhéneuc,et al.  A systematic literature review on the usage of eye-tracking in software engineering , 2015, Inf. Softw. Technol..

[20]  Bonita Sharif,et al.  iTrace: enabling eye tracking on software artifacts within the IDE to support software engineering tasks , 2015, ESEC/SIGSOFT FSE.

[21]  Thomas Fritz,et al.  Tracing software developers' eyes and interactions for change tasks , 2015, ESEC/SIGSOFT FSE.

[22]  Andrew Begel,et al.  Eye Movements in Code Reading: Relaxing the Linear Order , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[23]  Maarten A. Hogervorst,et al.  Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload , 2014, Front. Neurosci..

[24]  Mária Bieliková,et al.  Estimation of student's programming expertise , 2014, ESEM '14.

[25]  Andrew Begel,et al.  Using psycho-physiological measures to assess task difficulty in software development , 2014, ICSE.

[26]  Collin McMillan,et al.  Improving automated source code summarization via an eye-tracking study of programmers , 2014, ICSE.

[27]  Roman Bednarik,et al.  What influences dwell time during source code reading?: analysis of element type and frequency as factors , 2014, ETRA.

[28]  Nergiz Ercil Cagiltay,et al.  Performing and analyzing non-formal inspections of entity relationship diagram (ERD) , 2013, J. Syst. Softw..

[29]  Yann-Gaël Guéhéneuc,et al.  An empirical study on the efficiency of graphical vs. textual representations in requirements comprehension , 2013, 2013 21st International Conference on Program Comprehension (ICPC).

[30]  Yann-Gaël Guéhéneuc,et al.  Professional status and expertise for UML class diagram comprehension: An empirical study , 2012, 2012 20th IEEE International Conference on Program Comprehension (ICPC).

[31]  Sven Apel,et al.  Measuring programming experience , 2012, 2012 20th IEEE International Conference on Program Comprehension (ICPC).

[32]  Megan H. Papesh,et al.  Pupil Dilation Reflects the Creation and Retrieval of Memories , 2012 .

[33]  Roman Bednarik,et al.  Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations , 2012, Int. J. Hum. Comput. Stud..

[34]  Andreas Busjahn,et al.  Analysis of code reading to gain more insight in program comprehension , 2011, Koli Calling.

[35]  Raimund Dachselt,et al.  Using background colors to support program comprehension in software product lines , 2011, EASE.

[36]  B. Tversky,et al.  Effects of visual and verbal presentation on cognitive load in vigilance, memory, and arithmetic tasks. , 2011, Psychophysiology.

[37]  Yann-Gaël Guéhéneuc,et al.  An empirical study on the efficiency of different design pattern representations in UML class diagrams , 2010, Empirical Software Engineering.

[38]  Jeff Klingner,et al.  Fixation-aligned pupillary response averaging , 2010, ETRA.

[39]  Andrew L. Kun,et al.  Estimating cognitive load using remote eye tracking in a driving simulator , 2010, ETRA.

[40]  Marco Torchiano,et al.  The Role of Experience and Ability in Comprehension Tasks Supported by UML Stereotypes , 2007, 29th International Conference on Software Engineering (ICSE'07).

[41]  Tore Dybå,et al.  Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise , 2007, IEEE Transactions on Software Engineering.

[42]  Akito Monden,et al.  Analyzing individual performance of source code review using reviewers' eye movement , 2006, ETRA.

[43]  Markku Tukiainen,et al.  An eye-tracking methodology for characterizing program comprehension processes , 2006, ETRA.

[44]  Brian P. Bailey,et al.  Categories & Subject Descriptors: H.5.2 [Information , 2022 .

[45]  E. Granholm,et al.  Pupillometric measures of cognitive and emotional processes. , 2004, International journal of psychophysiology : official journal of the International Organization of Psychophysiology.

[46]  S. P. Marshall,et al.  The Index of Cognitive Activity: measuring cognitive workload , 2002, Proceedings of the IEEE 7th Conference on Human Factors and Power Plants.

[47]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[48]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[49]  W. Levelt,et al.  Pupillary dilation as a measure of attention: a quantitative system analysis , 1993 .

[50]  J. Shanteau.  Competence in experts: The role of task characteristics , 1992 .

[51]  Iris Vessey,et al.  Expertise in Debugging Computer Programs: A Process Analysis , 1984, Int. J. Man Mach. Stud..

[52]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[53]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982, Psychological bulletin.

[54]  M. Holland,et al.  Blinking and Thinking , 1975, Perceptual and motor skills.

[55]  Philip H. Ramsey.  Nonparametric Statistical Methods , 1974, Technometrics.

[56]  M. Holland,et al.  Blinking and Mental Load , 1972, Psychological reports.

[57]  D Kahneman,et al.  Pupil Diameter and Load on Memory , 1966, Science.

[58]  S. Sutton,et al.  Pupillary Response at Visual Threshold , 1966, Nature.

[59]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[60]  C. W. Telford,et al.  Some factors influencing voluntary and reflex eyelid responses , 1933 .

[61]  Dimosthenis Kontogiorgos,et al.  Towards identifying programming expertise with the use of physiological measures , 2015 .

[62]  Brian P. Bailey,et al.  Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management , 2008, TCHI.

[63]  Andrew T. Duchowski,et al.  Eye Tracking Methodology: Theory and Practice , 2003, Springer London.

[64]  Keith S. Karn,et al.  Commentary on Section 4. Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. , 2003 .

[65]  J. Beatty,et al.  The pupillary system. , 2000 .

[66]  Martha E. Crosby,et al.  How do we read algorithms? A case study , 1990, Computer.