Evaluating students’ understanding of statics concepts using eye gaze data

In engineering courses, exams and homework assignments are among the standard tools used to assess students’performance and comprehension of course material. However, they do not always provide opportunities to reveal whetherstudents truly understand related engineering concepts. This paper seeks to bridge that research gap by using eye-trackingtechnology to observe how students solve statics problems. In a within-subject experiment, twenty participants were askedto solve nine statics problems shown on a computer display. A non-invasive eye-tracker was used to record participants’eye movements during the problem solving process. Participants were then asked to explain how they solved threerepresentative problems. The results show that different eye gaze patterns exist between those who solved problemscorrectly and those who solved them incorrectly. For the specific concepts involved in solving these problems, those whocorrectly understood the concepts also exhibited different eye gaze patterns than those who did not. We also found thatstudents’ spatial visualization skills positively correlate with their performance when solving statics problems. Thisinvestigation showed that eye gaze data has the potential to serve as a diagnostic tool to discern how students solve staticsproblems and understand related engineering concepts. These results may provide insight into students’ problem-solvingstrategies and difficulties, and help instructors choose more adaptive teaching methods for students.

[1]  Sam Leinster,et al.  Some techniques for computer-based assessment in medical education , 1998 .

[2]  M. Scheerer,et al.  Problem Solving , 1967, Nature.

[3]  Theodore Branoff,et al.  Spatial Visualization Measurement: A Modification of the Purdue Spatial Visualization Test - Visualization of Rotations. , 2000 .

[4]  Andreas Glöckner,et al.  An eye‐tracking study on information processing in risky decisions: Evidence for compensatory strategies based on automatic processes , 2011 .

[5]  Michael J. Prince,et al.  Influence of engineering instructors' teaching and learning beliefs on pedagogies in engineering science courses , 2013 .

[6]  Dongheng Li,et al.  openEyes: a low-cost head-mounted eye-tracking solution , 2006, ETRA.

[7]  Stephanie M. Stalinski,et al.  Journal of Experimental Psychology: Learning, Memory, and Cognition , 2012 .

[8]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[9]  M. Posner,et al.  Attention and the detection of signals. , 1980, Journal of experimental psychology.

[10]  T. Crawford,et al.  Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection. , 2010, Journal of experimental psychology. Applied.

[11]  Adrian M. Madsen.  Studies of Visual Attention in Physics Problem Solving. , 2013 .

[12]  Andreas Bulling,et al.  Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction , 2014, UbiComp Adjunct.

[13]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[14]  John D. Gould,et al.  Field of View and Target Uncertainty in Visual Search and Inspection , 1973 .

[15]  Brianno Coller,et al.  An experiment in hands-on learning in engineering mechanics: Statics , 2008 .

[16]  R. Felder,et al.  Applications, Reliability and Validity of the Index of Learning Styles* , 2005 .

[17]  Huei-Tse Hou,et al.  Visual attention for solving multiple-choice science problem: An eye-tracking analysis , 2012, Comput. Educ..

[18]  Jean Underwood,et al.  Visual attention while driving: sequences of eye fixations made by experienced and novice drivers , 2003, Ergonomics.

[19]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[20]  Jennifer Robertson-Honecker,et al.  Online Homework, Help or Hindrance? What Students Think and How They Perform. , 2011 .

[21]  Tzyy-Ping Jung,et al.  Eye movements predict students' computer-based assessment performance of physics concepts in different presentation modalities , 2014, Comput. Educ..

[22]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[23]  Eyal M. Reingold,et al.  Eye movements reveal solution knowledge prior to insight , 2011, Consciousness and Cognition.

[24]  Dan Budny,et al.  Assessment of the Impact of Freshman Engineering Courses * , 1998 .

[25]  A. Glöckner,et al.  Multiple-reason decision making based on automatic processing. , 2008, Journal of experimental psychology. Learning, memory, and cognition.

[26]  Eric J. Johnson,et al.  The adaptive decision maker , 1993 .

[27]  N. Mackworth,et al.  Cognitive determinants of fixation location during picture viewing. , 1978, Journal of experimental psychology. Human perception and performance.

[28]  John Paulin Hansen,et al.  Evaluation of a low-cost open-source gaze tracker , 2010, ETRA.

[29]  R. Sternberg,et al.  Complex Problem Solving : Principles and Mechanisms , 1992 .

[30]  Ruth A. Streveler,et al.  Validating of the diagnostic capabilities of concept inventories: Preliminary evidence from the Concept Assessment Tool for Statics (CATS) , 2012 .

[31]  Erin F. MacDonald,et al.  Eye-Tracking Data Predict Importance of Product Features and Saliency of Size Change , 2014 .

[32]  Phil Davies,et al.  Computerized Peer Assessment , 2000 .

[33]  Paul S. Steif,et al.  Initial Data from a Statics Concept Inventory , 2004 .

[34]  Ronald H. Stevens,et al.  An Assessment of the Effect of Collaborative Groups on Students' Problem-Solving Strategies and Abilities. , 2008 .

[35]  R. Säljö,et al.  Expertise Differences in the Comprehension of Visualizations: a Meta-Analysis of Eye-Tracking Research in Professional Domains , 2011 .

[36]  E. Corte,et al.  Solving Compare Problems: An Eye-Movement Test of Lewis and Mayer's Consistency Hypothesis , 2007 .

[37]  Silvia Wen-Yu Lee,et al.  A review of using eye-tracking technology in exploring learning from 2000 to 2012 , 2013 .

[38]  Jose P. Mestre,et al.  Eye-Gaze Patterns as Students Study Worked-out Examples in Mechanics. , 2010 .

[39]  Paul S. Steif,et al.  Learning Modules for Statics , 2006 .

[40]  Katharina Scheiter,et al.  Attention guidance during example study via the model's eye movements , 2009, Comput. Hum. Behav..

[41]  M. Hegarty,et al.  Comprehension of Arithmetic Word Problems: Evidence from Students' Eye Fixations. , 1992 .

[42]  Joseph C. Hickox,et al.  Eye Scan Patterns of Experienced and Novice Pilots during Visual Flight Rules (VFR) Navigation , 1999 .

[43]  So Yoon Yoon,et al.  Psychometric properties of the Revised Purdue Spatial Visualization Tests: Visualization of Rotations (the Revised PSVT:R) , 2011 .

[44]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[45]  Matthew W. Lewis,et al.  Self-Explonations: How Students Study and Use Examples in Learning to Solve Problems , 1989, Cogn. Sci..

[46]  K. Scouller.  The influence of assessment method on students' learning approaches: Multiple choice question examination versus assignment essay , 1998 .

[47]  Katherine M. Steele,et al.  Feedback from in-class worksheets and discussion improves performance on the statics concept inventory , 2014 .

[48]  R. Felder,et al.  Learning and Teaching Styles in Engineering Education. , 1988 .

[49]  D. Hargreaves.  Student Learning and Assessment Are Inextricably Linked , 1997 .

[50]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[51]  Anand K. Gramopadhye,et al.  Gaze-augmented think-aloud as an aid to learning , 2012, CHI.

[52]  H. Montgomery,et al.  A think aloud study of dominance structuring in decision processes , 1989 .

[53]  Arthur M. Jacobs,et al.  OGAMA (Open Gaze and Mouse Analyzer): Open-source software designed to analyze eye and mouse movements in slideshow study designs , 2008, Behavior research methods.