Measuring cognitive load during simulation-based psychomotor skills training: sensitivity of secondary-task performance and subjective ratings

As interest in applying cognitive load theory (CLT) to the study and design of pedagogic and technological approaches in healthcare simulation grows, suitable measures of cognitive load (CL) are needed. Here, we report a two-phased study investigating the sensitivity of subjective ratings of mental effort (SRME) and secondary-task performance (signal detection rate, SDR and recognition reaction time, RRT) as measures of CL. In phase 1 of the study, novice learners and expert surgeons attempted a visual-monitoring task under two conditions: single-task (monitoring a virtual patient’s heart-rate) and dual-task (tying surgical knots on a bench-top simulator while monitoring the virtual patient’s heart-rate). Novices demonstrated higher mental effort and inferior secondary-task performance on the dual-task compared to experts (RRT 1.76 vs. 0.73, p = 0.012; SDR 0.27 vs. 0.97, p < 0.001; SRME 7.75 vs. 2.80, p < 0.001). Similarly, secondary task performance deteriorated from baseline to dual-task among novices (RRT 0.63 vs. 1.76 s, p < 0.006 and SDR 1.00 vs. 0.27, p < 0.001), but not experts (RRT 0.63 vs. 0.73 s, p = 0.124 and SDR 1.00 vs. 0.97, p = 0.178). In phase 2, novices practiced surgical knot-tying on the bench top simulator during consecutive dual-task trials. A significant increase in SDR (F(9,63) = 6.63, p < 0.001, f = 0.97) and decrease in SRME (F(9,63) = 9.39, p < 0.001, f = 1.04) was observed during simulation training, while RRT did not change significantly (F(9,63) = 1.18, p < 0.32, f = 0.41). The results suggest subjective ratings and dual-task performance can be used to track changes in CL among novices, particularly in early phases of simulation-based skills training. The implications for measuring CL in simulation instructional design research are discussed.

[1]  Dimitrios Stefanidis,et al.  Do novices display automaticity during simulator training? , 2008, American journal of surgery.

[2]  R. Reznick,et al.  Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. , 2007, Surgery.

[3]  D. Leutner,et al.  Direct Measurement of Cognitive Load in Multimedia Learning , 2003 .

[4]  David Rojas,et al.  What we call what we do affects how we do it: a new nomenclature for simulation research in medical education , 2014, Advances in health sciences education : theory and practice.

[5]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[6]  Jason H. Szostek,et al.  Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis , 2013, Medical teacher.

[7]  Debra Nestel,et al.  Simulation for Learning and Teaching Procedural Skills: The State of the Science , 2011, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[8]  K. A. Ericsson,et al.  Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. , 2004, Academic medicine : journal of the Association of American Medical Colleges.

[9]  Slava Kalyuga.  Cognitive Load Theory: How Many Types of Load Does It Really Need? , 2011 .

[10]  R. Brydges,et al.  Do not teach me while I am working! , 2012, American journal of surgery.

[11]  Walter Schneider,et al.  Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. , 1977 .

[12]  R. Reznick,et al.  Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. , 2007, American journal of surgery.

[13]  David A Cook,et al.  Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. , 2011, JAMA.

[14]  Krista E. DeLeeuw,et al.  A Comparison of Three Measures of Cognitive Load: Evidence for Separable Measures of Intrinsic, Extraneous, and Germane Load , 2008 .

[15]  James R Korndorffer,et al.  Redefining simulator proficiency using automaticity theory. , 2007, American journal of surgery.

[16]  Berit Eika,et al.  Effects of modelling examples in complex procedural skills training: a randomised study , 2013, Medical education.

[17]  Katharina Scheiter,et al.  Explaining the split-attention effect: Is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? , 2009, Comput. Hum. Behav..

[18]  H. Schmidt,et al.  Description, justification and clarification: a framework for classifying the purposes of research in medical education , 2008, Medical education.

[19]  F. Paas.  Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. , 1992 .

[20]  Irene W. Y. Ma,et al.  Emotion, cognitive load and learning outcomes during simulation training , 2012, Medical education.

[21]  W. B. Seales,et al.  Assessing Mental Workload During Laparoscopic Surgery , 2005, Surgical innovation.

[22]  A. Byrne,et al.  Novel method of measuring the mental workload of anaesthetists during clinical practice. , 2009, British journal of anaesthesia.

[23]  M. Mon-Williams,et al.  Motor Control and Learning , 2006 .

[24]  John Sweller,et al.  Cognitive Load During Problem Solving: Effects on Learning , 1988, Cogn. Sci..

[25]  T. Gog,et al.  Development of an instrument for measuring different types of cognitive load , 2013, Behavior Research Methods.

[26]  F. Paas,et al.  Cognitive Load Theory and Instructional Design: Recent Developments , 2003 .

[27]  J. Barsuk,et al.  Does Simulation-Based Medical Education With Deliberate Practice Yield Better Results Than Traditional Clinical Education? A Meta-Analytic Comparative Review of the Evidence , 2011, Academic medicine : journal of the Association of American Medical Colleges.

[28]  Adam Dubrowski,et al.  Surgical skill acquisition with self-directed practice using computer-based video training. , 2007, American journal of surgery.

[29]  T. Jong.  Cognitive load theory, educational research, and instructional design: some food for thought , 2010 .

[30]  R. Magill.  Motor Learning And Control , 1980 .

[31]  F. Paas,et al.  Memory load and the cognitive pupillary response in aging. , 2004, Psychophysiology.

[32]  D. Leutner,et al.  Assessment of cognitive load in multimedia learning using dual-task methodology. , 2002, Experimental psychology.

[33]  Paul Ayres.  Using subjective measures to detect variations of intrinsic cognitive load within problems , 2006 .

[34]  John Q. Young,et al.  Cognitive Load Theory: Implications for medical education: AMEE Guide No. 86 , 2014, Medical teacher.

[35]  A. Darzi,et al.  The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. , 2001, Journal of the American College of Surgeons.

[36]  M. Just,et al.  Interdependence of Nonoverlapping Cortical Systems in Dual Cognitive Tasks , 2001, NeuroImage.

[37]  Charlotte Ringsted,et al.  Setting a research agenda for simulation-based healthcare education: a synthesis of the outcome from an Utstein style meeting. , 2011, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[38]  F. Paas,et al.  Measurement of Cognitive Load in Instructional Research , 1994, Perceptual and motor skills.

[39]  J. Sweller,et al.  Cognitive load theory in health professional education: design principles and strategies , 2010, Medical education.

[40]  R. Shiffrin,et al.  Controlled and automatic human information processing: I , 1977 .

[41]  H. Macrae,et al.  Technical skill training improves the ability to learn. , 2011, Surgery.

[42]  F. Paas,et al.  Cognitive Load Measurement as a Means to Advance Cognitive Load Theory , 2003 .

[43]  Adam Dubrowski,et al.  Computer-assisted assessment of one-handed knot tying skills performed within various contexts: a construct validity study. , 2006, American journal of surgery.

[44]  Pieter Wouters,et al.  Learner Performance in Multimedia Learning Arrangements: An Analysis Across Instructional Approaches , 2009 .

[45]  F. Paas,et al.  Cognitive Architecture and Instructional Design , 1998 .

[46]  Anique B. H. de Bruin,et al.  Checklists improve experts’ diagnostic decisions , 2013, Medical education.

[47]  Pavlo D. Antonenko,et al.  Using Electroencephalography to Measure Cognitive Load , 2010 .

[48]  R. Aggarwal,et al.  Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery , 2008, The British journal of surgery.

[49]  J. Sweller.  COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[50]  Adam Dubrowski,et al.  The Impact of Secondary-Task Type on the Sensitivity of Reaction-Time Based Measurement of Cognitive Load for Novices Learning Surgical Skills Using Simulation , 2014, MMVR.

[51]  F. Paas,et al.  Variability of Worked Examples and Transfer of Geometrical Problem-Solving Skills: A Cognitive-Load Approach , 1994 .

[52]  F. Paas,et al.  Effects of the Physical Environment on Cognitive Load and Learning: Towards a New Model of Cognitive Load , 2014 .