Tracking Students’ Cognitive Processes During Program Debugging—An Eye-Movement Approach

This study explores students' cognitive processes while debugging programs by using an eye tracker. Students' eye movements during debugging were recorded by an eye tracker to investigate whether and how high- and low-performance students act differently during debugging. Thirty-eight computer science undergraduates were asked to debug two C programs. The path of students' gaze while following program codes was subjected to sequential analysis to reveal significant sequences of areas examined. These significant gaze path sequences were then compared to those of students with different debugging performances. The results show that, when debugging, high-performance students traced programs in a more logical manner, whereas low-performance students tended to stick to a line-by-line sequence and were unable to quickly derive the program's higher-level logic. Low-performance students also often jumped directly to certain suspected statements to find bugs, without following the program's logic. They also often needed to trace back to prior statements to recall information, and spent more time on manual computation. Based on the research results, adaptive instructional strategies and materials can be developed for students of different performance levels, to improve associated cognitive activities during debugging, which can foster learning during debugging and programming.

[1]  Roman Bednarik,et al.  Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations , 2012, Int. J. Hum. Comput. Stud..

[2]  Andrew Begel,et al.  Eye Movements in Code Reading: Relaxing the Linear Order , 2015, 2015 IEEE 23rd International Conference on Program Comprehension.

[3]  Camilo Jiménez,et al.  Developing programming skills by using interactive learning objects , 2009, ITiCSE '09.

[4]  N. Mackintosh,et al.  The fractionation of working memory maps onto different components of intelligence , 2003 .

[5]  Jukka Hyönä,et al.  The Use of Eye Movements in the Study of Multimedia Learning. , 2010 .

[6]  Robert S. Rist Program Structure and Design , 1995, Cogn. Sci..

[7]  T. Gog,et al.  In the eyes of the beholder: How experts and novices interpret dynamic stimuli , 2010 .

[8]  Jonathan I. Maletic,et al.  An eye-tracking study on the role of scan time in finding source code defects , 2012, ETRA.

[9]  Andrew Begel,et al.  Eye tracking in computing education , 2014, ICER '14.

[10]  Ruven E. Brooks Towards a theory of the cognitive processes in computer programming , 1999, Int. J. Hum. Comput. Stud..

[11]  Sue Fitzgerald,et al.  Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers , 2008, Comput. Sci. Educ..

[12]  Collin McMillan,et al.  Improving automated source code summarization via an eye-tracking study of programmers , 2014, ICSE.

[13]  Markku Tukiainen,et al.  Temporal eye-tracking data: evolution of debugging strategies with multiple representations , 2008, ETRA.

[14]  Derek H. Sleeman,et al.  A Summary of Misconceptions of High School Basic Programmers , 1984 .

[15]  D. Janaki Ram,et al.  Identification of Cognitive Processes of Effective and Ineffective Students During Computer Programming , 2010, TOCE.

[16]  K. Rayner,et al.  Eye Movements as Reflections of Comprehension Processes in Reading , 2006 .

[17]  Anneliese Amschler Andrews,et al.  Identification of Dynamic Comprehension Processes During Large Scale Maintenance , 1996, IEEE Trans. Software Eng..

[18]  Gary Lewandowski,et al.  Debugging: the good, the bad, and the quirky -- a qualitative analysis of novices' strategies , 2008, SIGCSE '08.

[19]  Fred Paas,et al.  Uncovering cognitive processes: Different techniques that can contribute to cognitive load research and instruction , 2009, Comput. Hum. Behav..

[20]  Anthony F. Norcio,et al.  The effects of beacons, comments, and tasks on program comprehension process in software maintenance , 2010 .

[21]  John D. Gould,et al.  An Exploratory Study of Computer Program Debugging1 , 1974 .

[22]  Hsueh-Chih Chen,et al.  Eye tracking technology for learning and education , 2010 .

[23]  Michael J. Morgan,et al.  Learning challenges faced by novice programming students studying high level and low feedback concepts , 2007 .

[24]  Luisa M. Regueras,et al.  A distributed system for learning programming on-line , 2012, Comput. Educ..

[25]  Gunnar R. Bergersen,et al.  Programming Skill, Knowledge, and Working Memory Among Professional Software Developers from an Investment Theory Perspective , 2011 .

[26]  Jim Buckley,et al.  Inference-based and expectation-based processing in program comprehension , 2001, Proceedings 9th International Workshop on Program Comprehension. IWPC 2001.

[27]  Bonita Sharif,et al.  An eye-tracking study assessing the comprehension of c++ and Python source code , 2014, ETRA.

[28]  Randall W Engle,et al.  Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. , 1999, Journal of experimental psychology. General.

[29]  Simon P. Davies,et al.  Models and Theories of Programming Strategy , 1993, Int. J. Man Mach. Stud..

[30]  Ben Shneiderman,et al.  Syntactic/semantic interactions in programmer behavior: A model and experimental results , 1979, International Journal of Computer & Information Sciences.

[31]  F. Paas,et al.  Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. , 2005, Journal of experimental psychology. Applied.

[32]  Mark Weiser,et al.  Programmers use slices when debugging , 1982, CACM.

[33]  Iain Milne,et al.  Difficulties in Learning and Teaching Programming—Views of Students and Tutors , 2002, Education and Information Technologies.

[34]  Philip Smith Roger Bakeman John M. Gottman , 1987, Animal Behaviour.

[35]  M. Crosby,et al.  Code Scanning Patterns in Program Comprehension , 2005 .

[36]  Ruven E. Brooks,et al.  Towards a Theory of the Comprehension of Computer Programs , 1983, Int. J. Man Mach. Stud..

[37]  Kate Ehrlich,et al.  Empirical Studies of Programming Knowledge , 1984, IEEE Transactions on Software Engineering.

[38]  Akito Monden,et al.  Analyzing individual performance of source code review using reviewers' eye movement , 2006, ETRA.

[39]  Shaochun Xu,et al.  Cognitive process during program debugging , 2004, Proceedings of the Third IEEE International Conference on Cognitive Informatics, 2004..

[40]  Sheng-Yi Wu,et al.  Analyzing the social knowledge construction behavioral patterns of an online synchronous collaborative discussion instructional activity using an instant messaging tool: A case study , 2011, Comput. Educ..

[41]  Veranika Lim,et al.  Eye Gaze and Mouse Cursor Relationship in a Debugging Task , 2013, HCI.

[42]  Gwo-Jen Hwang,et al.  The influences of a two-tier test strategy on student learning: A lag sequential analysis approach , 2015, Comput. Educ..

[43]  Markku Tukiainen,et al.  An eye-tracking methodology for characterizing program comprehension processes , 2006, ETRA.

[44]  Martha E. Crosby,et al.  How do we read algorithms? A case study , 1990, Computer.

[45]  Jayaprakash S,et al.  Classification of Cognitive Difficulties of Students to Learn Computer Programming , 2009 .

[46]  Jean Scholtz,et al.  The Roles Beacons Play in Comprehension for Novice and Expert Programmers , 2002, PPIG.

[47]  C Bron,et al.  COGNITIVE STRATEGIES AND LOOPING CONSTRUCTS - AN EMPIRICAL-STUDY , 1984 .

[48]  Andreas Busjahn,et al.  Analysis of code reading to gain more insight in program comprehension , 2011, Koli Calling.

[49]  Elliot Soloway,et al.  A Goal/Plan Analysis of Buggy Pascal Programs , 1985, Hum. Comput. Interact..

[50]  Jawed I. A. Siddiqi,et al.  Specification Influences in Program Design , 1989, Int. J. Man Mach. Stud..

[51]  Huei-Tse Hou,et al.  Visual attention for solving multiple-choice science problem: An eye-tracking analysis , 2012, Comput. Educ..

[52]  Tanya McGill,et al.  A conceptual framework for analyzing students’ knowledge of programming , 1997 .

[53]  Susan Wiedenbeck,et al.  Beacons in Computer Program Comprehension , 1986, Int. J. Man Mach. Stud..

[54]  Renee Hobbs,et al.  Conditions of Learning in Novice Programmers , 1986 .

[55]  Matthew C. Jadud A First Look at Novice Compilation Behaviour Using BlueJ , 2005, Comput. Sci. Educ..

[56]  Silvia Wen-Yu Lee,et al.  A review of using eye-tracking technology in exploring learning from 2000 to 2012 , 2013 .

[57]  Robert McCartney,et al.  Common sense computing (episode 4): debugging , 2008, Comput. Sci. Educ..

[58]  Jennifer L Branch Investigating the Information-Seeking Processes of Adolescents: The Value of Using Think Alouds and Think Afters , 2000 .

[59]  G J Hitch,et al.  Working memory and children's mental addition. , 1997, Journal of experimental child psychology.