Technology-Enhanced Simulation to Assess Health Professionals: A Systematic Review of Validity Evidence, Research Methods, and Reporting Quality

Purpose To summarize the tool characteristics, sources of validity evidence, methodological quality, and reporting quality for studies of technology-enhanced simulation-based assessments for health professions learners. Method The authors conducted a systematic review, searching MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous reviews through May 2011. They selected original research in any language evaluating simulation-based assessment of practicing and student physicians, nurses, and other health professionals. Reviewers working in duplicate evaluated validity evidence using Messick’s five-source framework; methodological quality using the Medical Education Research Study Quality Instrument and the revised Quality Assessment of Diagnostic Accuracy Studies; and reporting quality using the Standards for Reporting Diagnostic Accuracy and Guidelines for Reporting Reliability and Agreement Studies. Results Of 417 studies, 350 (84%) involved physicians at some stage in training. Most focused on procedural skills, including minimally invasive surgery (N = 142), open surgery (81), and endoscopy (67). Common elements of validity evidence included relations with trainee experience (N = 306), content (142), relations with other measures (128), and interrater reliability (124). Of the 217 studies reporting more than one element of evidence, most were judged as having high or unclear risk of bias due to selective sampling (N = 192) or test procedures (132). Only 64% proposed a plan for interpreting the evidence to be presented (validity argument). Conclusions Validity evidence for simulation-based assessments is sparse and is concentrated within specific specialties, tools, and sources of validity evidence. The methodological and reporting quality of assessment studies leaves much room for improvement.

[1]  Sarah Garside,et al.  Method and reporting quality in health professions education research: a systematic review , 2011, Medical education.

[2]  A. Duffy,et al.  Computer-based endoscopy simulation: emerging roles in teaching and professional skills assessment. , 2008, Journal of surgical education.

[3]  Todd Dorman,et al.  The Reported Validity and Reliability of Methods for Evaluating Continuing Medical Education: A Systematic Review , 2008, Academic medicine : journal of the Association of American Medical Colleges.

[4]  Susan Mallett,et al.  QUADAS-2: A Revised Tool for the Quality Assessment of Diagnostic Accuracy Studies , 2011, Annals of Internal Medicine.

[5]  A. Hrõbjartsson,et al.  Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. , 2011, Journal of clinical epidemiology.

[6]  P. Bossuyt,et al.  Empirical evidence of design-related bias in studies of diagnostic tests. , 1999, JAMA.

[7]  S. Seiden,et al.  Patient Simulation: A Literary Synthesis of Assessment Tools in Anesthesiology , 2009, Journal of educational evaluation for health professions.

[8]  P. Bossuyt,et al.  Sources of Variation and Bias in Studies of Diagnostic Accuracy , 2004, Annals of Internal Medicine.

[9]  Mark Albanese,et al.  Perspective: Competency-based medical education: a defense against the four horsemen of the medical education apocalypse. , 2008, Academic medicine : journal of the Association of American Medical Colleges.

[10]  M. Roizen,et al.  Technology-enhanced simulation for health professions education: a systematic review and meta-analysis , 2012 .

[11]  N. Powe,et al.  A Systematic Review of the Methodological Rigor of Studies Evaluating Cultural Competence Training of Health Professionals , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[12]  E. Emanuel,et al.  Shortening medical training by 30%. , 2012, JAMA.

[13]  David Moher,et al.  Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. , 2004, Family practice.

[14]  J. D. Greaves,et al.  Assessment instruments used during anaesthetic simulation: review of published studies. , 2001, British journal of anaesthesia.

[15]  Piotr Maciej Skorupiński.  American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Standards for educational and psychological testing , 2015 .

[16]  Joann G Elmore,et al.  Trends in study methods used in undergraduate medical education research, 1969-2007. , 2007, JAMA.

[17]  Aimin Zhang,et al.  Construct validity testing of a laparoscopic surgery simulator (Lap Mentor®) , 2008, Surgical Endoscopy.

[18]  A. Scherpbier,et al.  Validation and implementation of surgical simulators: a critical review of present, past, and future , 2009, Surgical Endoscopy.

[19]  T. Habermann,et al.  Association of resident fatigue and distress with perceived medical errors. , 2009, JAMA.

[20]  David A Cook,et al.  Quality of reporting of experimental studies in medical education: a systematic review , 2007, Medical education.

[21]  S. Downing.  Validity: on the meaningful interpretation of assessment data , 2003, Medical education.

[22]  David A Cook,et al.  Association between funding and quality of published medical education research. , 2007, JAMA.

[23]  K. Adamson,et al.  A Review of Currently Published Evaluation Instruments for Human Patient Simulation , 2010 .

[24]  David A. Cook,et al.  Predictive Validity Evidence for Medical Education Research Study Quality Instrument Scores: Quality of Submissions to JGIM’s Medical Education Special Issue , 2008, Journal of General Internal Medicine.

[25]  Jonathan Sherbino,et al.  The role of assessment in competency-based medical education , 2010, Medical teacher.

[26]  A. Duffy,et al.  Construct validity for the LAPSIM laparoscopic surgical simulator , 2005, Surgical Endoscopy And Other Interventional Techniques.

[27]  A. Ziv,et al.  Simulation-based medical education: an ethical imperative. , 2006, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[28]  W. Gantert,et al.  Can skills assessment on a virtual reality trainer predict a surgical trainee’s talent in laparoscopic surgery? , 2006, Surgical Endoscopy And Other Interventional Techniques.

[29]  R. Reznick,et al.  Objective structured assessment of technical skill (OSATS) for surgical residents , 1997, The British journal of surgery.

[30]  David L Waldman,et al.  Use of computer simulation for determining endovascular skill levels in a carotid stenting model. , 2004, Journal of vascular surgery.

[31]  Vadim Sherman,et al.  Using simulators to assess laparoscopic competence: ready for widespread use? , 2004, Surgery.

[32]  A. Darzi,et al.  The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. , 2001, Journal of the American College of Surgeons.

[33]  Ara Darzi,et al.  Effectiveness of procedural simulation in urology: a systematic review. , 2011, The Journal of urology.

[34]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[35]  Brian E Clauser,et al.  Validity considerations in the assessment of professionalism , 2012, Advances in health sciences education : theory and practice.

[36]  A. Pereira,et al.  Competency-Based Education and Training in Internal Medicine , 2010, Annals of Internal Medicine.

[37]  Marlies P Schijven,et al.  ProMIS Augmented Reality Training of Laparoscopic Procedures Face Validity , 2008, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[38]  Lisa D Howley,et al.  Quality of standardised patient research reports in the medical education literature: review and recommendations , 2008, Medical education.

[39]  D. Farley,et al.  ACGME duty-hour recommendations - a national survey of residency program directors. , 2010, The New England journal of medicine.

[40]  M. Humayun,et al.  VIRTUAL VITREORETINAL SURGICAL SIMULATOR AS A TRAINING TOOL , 2004, Retina.

[41]  Mark A Albanese,et al.  Defining characteristics of educational competencies , 2008, Medical education.

[42]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[43]  K. Horvath,et al.  Methodologies for establishing validity in surgical simulation studies. , 2010, Surgery.

[44]  A. Gallagher,et al.  Objective psychomotor skills assessment of experienced and novice flexible endoscopists with a virtual reality simulator , 2003, Journal of Gastrointestinal Surgery.

[45]  D. Rennie,et al.  Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative , 2003, BMJ : British Medical Journal.

[46]  Eric S. Holmboe,et al.  Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. , 2009, JAMA.

[47]  G. Fried,et al.  Development of a model for training and evaluation of laparoscopic skills. , 1998, American journal of surgery.

[48]  D. Cook,et al.  What is the validity evidence for assessments of clinical teaching? , 2005, Journal of General Internal Medicine.

[49]  D. Cook,et al.  Current concepts in validity and reliability for psychometric instruments: theory and application. , 2006, The American journal of medicine.

[50]  E. Pahle,et al.  Virtual reality computer simulation , 2001, Surgical Endoscopy.