How prior knowledge affects problem-solving performance in a medical simulation game: Using game-logs and eye-tracking

Abstract Computer-based simulation games provide an environment to train complex problem-solving skills. Yet, it is largely unknown how the in-game performance of learners varies with different levels of prior knowledge. Based on theories of complex-skill acquisition (e.g., 4C/ID), we derive four performance aspects that prior knowledge may affect: (1) systematicity in approach, (2) accuracy in visual attention and motor reactions, (3) speed in performance, and (4) cognitive load. This study aims to empirically test whether prior knowledge affects these four aspects of performance in a medical simulation game for resuscitation skills training. Participants were 24 medical professionals (experts, with high prior knowledge) and 22 medical students (novices, with low prior knowledge). After pre-training, they all played one scenario, during which game-logs and eye-movements were collected. A cognitive-load questionnaire ensued. During game play, experts demonstrated a more systematic approach, higher accuracy in visual selection and motor reaction, and a higher performance speed than novices. Their reported levels of cognitive load were lower. These results indicate that prior knowledge has a substantial impact on performance in simulation games, opening up the possibility of using our measures for performance assessment.

[1]  Huei-Tse Hou,et al.  Integrating cluster and sequential analysis to explore learners' flow and behavioral patterns in a simulation game with situated-learning context for science courses: A video-based process exploration , 2015, Comput. Hum. Behav..

[2]  G R Norman,et al.  The risks of thoroughness: Reliability and validity of global ratings and checklists in an OSCE , 1996, Advances in health sciences education : theory and practice.

[3]  L. Baum,et al.  A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains , 1970 .

[4]  Frank J. Lee,et al.  Does Learning a Complex Task Have to Be Complex?: A Study in Learning Decomposition , 2001, Cognitive Psychology.

[5]  Paul A. Kirschner,et al.  Ten Steps to Complex Learning: A Systematic Approach to Four-Component Instructional Design , 2007 .

[6]  P. Suppes,et al.  A model of eye movements and visual working memory during problem solving in geometry , 2001, Vision Research.

[7]  John C. Nesbit,et al.  Sequential Pattern Analysis of Learning Logs: Methodology and Applications , 2010 .

[8]  Günter Wallner,et al.  Visualization-based analysis of gameplay data - A review of literature , 2013, Entertain. Comput..

[9]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[10]  Wen Qu,et al.  Using gameplay data to examine learning behavior patterns in a serious game , 2017, Comput. Hum. Behav..

[11]  Thomas J. Palmeri,et al.  Theories of automaticity and the power law of practice. , 1999 .

[12]  Katie Salen.  The ecology of games : connecting youth, games, and learning , 2008 .

[13]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[14]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[15]  K. J. Williams,et al.  Quantitative description of the workload associated with airway management procedures. , 2000, Journal of clinical anesthesia.

[16]  John Sweller,et al.  Teaching general problem-solving skills is not a substitute for, or a viable addition to, teaching mathematics , 2010 .

[17]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[18]  Kenneth Holmqvist,et al.  Eye tracking: a comprehensive guide to methods and measures , 2011 .

[19]  Cheryl I. Johnson,et al.  An eye movement analysis of the spatial contiguity effect in multimedia learning. , 2012, Journal of experimental psychology. Applied.

[20]  Laura M. Naismith,et al.  Cognitive load predicts point-of-care ultrasound simulator performance , 2018, Perspectives on Medical Education.

[21]  F. Paas,et al.  Cognitive Architecture and Instructional Design: 20 Years Later , 2019, Educational Psychology Review.

[22]  Alexandre Perera-Lluna,et al.  Applying learning analytics to students' interaction in business simulation games. The usefulness of learning analytics to know what students really learn , 2019, Comput. Hum. Behav..

[23]  S. Liversedge,et al.  Oxford handbook of eye movements , 2011 .

[24]  F. Gobet,et al.  The Cambridge handbook of expertise and expert performance , 2006 .

[25]  Damien Litchfield,et al.  Looking for Cancer: Expertise Related Differences in Searching and Decision Making , 2013 .

[26]  Sandra P. Marshall,et al.  Eye tracking insights into cognitive modeling , 2006, ETRA.

[27]  Karl F. Van Orden,et al.  Assessment of Variable Coded Symbology Using Visual Search Performance and Eye Fixation Measures , 1998 .

[28]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[29]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[30]  K. Holmqvist,et al.  Reading Information Graphics: the Role of Spatial Contiguity and Dual Attentional Guidance , 2022 .

[31]  Florian Schmidt-Weigand,et al.  A closer look at split visual attention in system- and self-paced instruction in multimedia learning , 2010 .

[32]  Jeroen J. G. van Merriënboer,et al.  Perspectives on problem solving and instruction , 2013, Comput. Educ..

[33]  Douglas B. Clark,et al.  Driving Assessment of Students’ Explanations in Game Dialog Using Computer-Adaptive Testing and Hidden Markov Modeling , 2012 .

[34]  Asaf Degani,et al.  A MULTI-DIMENSIONAL EVALUATION METHODOLOGY FOR NEW COCKPIT SYSTEMS , 1999 .

[35]  Jeroen J. G. van Merrienboer,et al.  Assessing the Assessment in Emergency Care Training , 2014, PloS one.

[36]  Paul A. Kirschner,et al.  Identification of effective visual problem solving strategies in a complex visual domain , 2014 .

[37]  Orit Hazzan,et al.  Problem-solving strategies , 1986 .

[38]  Roland Brünken,et al.  Measurement of cognitive load in multimedia learning: a comparison of different objective measures , 2017 .

[39]  Rajesh P. N. Rao,et al.  PSYCHOLOGICAL SCIENCE Research Article EYE MOVEMENTS REVEAL THE SPATIOTEMPORAL DYNAMICS OE VISUAL SEARCH , 2022 .

[40]  Raymond Bond,et al.  Computing eye gaze metrics for the automatic assessment of radiographer performance during X-ray image interpretation , 2017, Int. J. Medical Informatics.

[41]  Thomas H. Hammond,et al.  Learning in Hierarchies , 2007 .

[42]  Sara de Freitas,et al.  Using games and simulations for supporting learning , 2006 .

[43]  R. Miall,et al.  A Painter's Eye Movements: A Study of Eye and Hand Movement during Portrait Drawing , 2001, Leonardo.

[44]  Elina Eriksson,et al.  Nursing students’ experiential learning processes using an online 3D simulation game , 2015, Education and Information Technologies.

[45]  Scott Makeig,et al.  Eye Activity Correlates of Workload during a Visuospatial Memory Task , 2001, Hum. Factors.

[46]  Dirk Ifenthaler,et al.  Serious Games Analytics: Theoretical Framework , 2015 .

[47]  F. Bartlett,et al.  Remembering: A Study in Experimental and Social Psychology , 1932 .

[48]  Jeroen J. G. van Merrienboer,et al.  An experimental study on the effects of a simulation game on students’ clinical cognitive skills and motivation , 2015, Advances in Health Sciences Education.

[49]  Joel S. Warm,et al.  Vigilance, Workload, and Stress , 2018 .

[50]  K. Squire.  Open-Ended Video Games: A Model for Developing Learning for the Interactive Age , 2007 .

[51]  James J. Gibson,et al.  The Ecological Approach to Visual Perception: Classic Edition , 2014 .

[52]  Roland Brünken,et al.  Emotional design and positive emotions in multimedia learning: An eyetracking study on the use of anthropomorphisms , 2015, Comput. Educ..

[53]  H. Haider,et al.  Eye movement during skill acquisition: More evidence for the information-reduction hypothesis , 1999 .

[54]  E. Reingold,et al.  Eye movements and visual expertise in chess and medicine , 2011 .

[55]  Kenneth Holmqvist,et al.  Threats to the validity of eye-movement research in psychology , 2018, Behavior research methods.

[56]  Jacob L. Orquin,et al.  Attention and choice: a review on eye movements in decision making. , 2013, Acta psychologica.

[57]  Roland Brünken,et al.  Learner characteristics and information processing in multimedia learning: A moderated mediation of the seductive details effect , 2016 .

[58]  T. Crawford,et al.  How do radiologists do it? The influence of experience and training on searching for chest nodules. , 2006 .

[59]  K. Kiili.  Digital game-based learning: Towards an experiential gaming model , 2005, Internet High. Educ..