Tracking the Decision-Making Process in Multiple-Choice Assessment: Evidence from Eye Movements

Summary This study investigated students' decision-making processes in a knowledge-assessing multiple-choice (MC) test using eye-tracking methodology. More precisely, the gaze bias effect (more attention to more preferred options) and its relation to domain knowledge were the focus of the study. Eye movements of students with high (HPK) and low (LPK) prior domain knowledge were recorded while they solved 21 MC items. Afterwards, students rated every answer option according to their subjective preference. As expected, both HPK and LPK students showed a gaze bias towards subjectively preferred answer options, whereby HPK students spent more time on objectively correct answers. Furthermore, a fine-grained time-course analysis showed similar patterns of attention distribution over time for both HPK and LPK students, when focusing on subjective preference levels. Thus, these data offer a new perspective on knowledge-related MC item solving and provide evidence for the generalizability of the gaze bias effect across decision tasks. Copyright © 2014 John Wiley & Sons, Ltd.

[1]  Eyal M Reingold,et al.  Evidence for top-down control of eye movements during visual decision making. , 2010, Journal of vision.

[2]  J. E. Russo,et al.  An Eye-Fixation Analysis of Choice Processes for Consumer Nondurables , 1994 .

[3]  Michael C. Rodriguez,et al.  A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment , 2002 .

[4]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[5]  Mary Hegarty,et al.  Thinking about the weather: How display salience and knowledge affect performance in a graphic inference task. , 2010, Journal of experimental psychology. Learning, memory, and cognition.

[6]  M. Posner,et al.  Attention and the detection of signals. , 1980, Journal of experimental psychology.

[7]  Jörg Wittwer,et al.  Pictures in Test Items: Effects on Response Time and Response Correctness , 2012 .

[8]  Ralf Engbert,et al.  Microsaccades uncover the orientation of covert attention , 2003, Vision Research.

[9]  Characterizing Visual Behaviour in a Lineup Task , 2009 .

[10]  M. Posner,et al.  Orienting of Attention* , 1980, The Quarterly journal of experimental psychology.

[11]  Huei-Tse Hou,et al.  Visual attention for solving multiple-choice science problem: An eye-tracking analysis , 2012, Comput. Educ..

[12]  Eyal M. Reingold,et al.  Eye Movement Monitoring as a Process Tracing Methodology in Decision Making Research , 2011 .

[13]  R. Pieters,et al.  Visual attention during brand choice : The impact of time pressure and task motivation , 1999 .

[14]  R. Säljö,et al.  Expertise Differences in the Comprehension of Visualizations: a Meta-Analysis of Eye-Tracking Research in Professional Domains , 2011 .

[15]  Joanna S. Gorin,et al.  Improving Construct Validity with Cognitive Psychology Principles. , 2001 .

[16]  Eyal M. Reingold,et al.  Eye movements reveal solution knowledge prior to insight , 2011, Consciousness and Cognition.

[17]  Gerald L. Lohse,et al.  A Comparison of Two Process Tracing Methods for Choice Tasks , 1996 .

[18]  N. J. Pienta,et al.  Eye-Tracking Study of Complexity in Gas Law Problems. , 2012 .

[19]  Frederick J. Brigham,et al.  An exploration of the use of eye‐gaze tracking to study problem‐solving on standardized science assessments , 2006 .

[20]  Mitsuru Ishizuka,et al.  AutoSelect: What You Want Is What You Get: Real-Time Processing of Visual Attention and Affect , 2006, PIT.

[21]  Ornella Robutti,et al.  A method for quantifying focused versus overview behavior in AOI sequences , 2011, Behavior research methods.

[22]  T. Mitsuda,et al.  Gaze bias during visual preference judgements: Effects of stimulus category and decision instructions , 2014 .

[23]  Heather D. Flowe,et al.  An exploration of visual behaviour in eyewitness identification tests , 2011 .

[24]  R. Nagel,et al.  Search Dynamics in Consumer Choice under Time Pressure: An Eye-Tracking Study , 2011 .

[25]  Eyal M. Reingold,et al.  The time course of gaze bias in visual decision tasks , 2009 .

[26]  Jukka Hyönä,et al.  The Use of Eye Movements in the Study of Multimedia Learning. , 2010 .

[27]  S. Shimojo,et al.  Early interactions between orienting, visual sampling and decision making in facial preference , 2006, Vision Research.

[28]  Mary Hegarty,et al.  Effects of knowledge and display design on comprehension of complex graphics , 2010 .

[29]  G. Cottrell,et al.  An examination of simultaneous lineup identification decision processes using eye tracking , 2011 .

[30]  Eyal M Reingold,et al.  Stimulus exposure and gaze bias: A further test of the gaze cascade model , 2009, Attention, perception & psychophysics.

[31]  D. Alais,et al.  Orientation tuning of contrast masking caused by motion streaks. , 2010, Journal of vision.

[32]  S. Shimojo,et al.  Gaze bias both reflects and influences preference , 2003, Nature Neuroscience.

[33]  Bob Rehder,et al.  How prior knowledge affects selective attention during category learning: An eyetracking study , 2011, Memory & cognition.

[34]  Gary D. Bird,et al.  The role of eye movements in decision making and the prospect of exposure effects , 2012, Vision Research.

[35]  D. Robinson,et al.  Shared neural control of attentional shifts and eye movements , 1996, Nature.

[36]  M A Just,et al.  A theory of reading: from eye fixations to comprehension. , 1980, Psychological review.

[37]  F. Paas,et al.  Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. , 2005, Journal of experimental psychology. Applied.

[38]  Paul D. Nichols,et al.  A Framework for Developing Cognitively Diagnostic Assessments , 1994 .

[39]  Jukka Hyönä,et al.  How prior knowledge, WMC, and relevance of information affect eye fixations in expository text. , 2003, Journal of experimental psychology. Learning, memory, and cognition.

[40]  Jacqueline P. Leighton.  Avoiding Misconception, Misuse, and Missed Opportunities: The Collection of Verbal Reports in Educational Achievement Testing , 2005 .

[41]  Eyal M. Reingold,et al.  Predicting preference from fixations , 2009, PsychNology J..

[42]  T. Gog,et al.  In the eyes of the beholder: How experts and novices interpret dynamic stimuli , 2010 .

[43]  T. Gog,et al.  Effects of prior knowledge and concept-map structure on disorientation, cognitive load, and learning , 2009 .

[44]  T. Gog,et al.  Uncovering Expertise-Related Differences in Troubleshooting Performance: Combining Eye Movement and Concurrent Verbal Protocol Data , 2005 .

[45]  S. Holm.  A Simple Sequentially Rejective Multiple Test Procedure , 1979 .

[46]  Elizabeth R Schotter,et al.  Gaze bias: Selective encoding and liking effects , 2010 .

[47]  A. Rangel,et al.  Multialternative drift-diffusion model predicts the relationship between visual fixations and choice in value-based decisions , 2011, Proceedings of the National Academy of Sciences.