Development of an Objective Scoring System for Measurement of Resident Performance on the Human Patient Simulator

Background:The decrease in the percentage of patients having cesarean delivery during general anesthesia has led some educators to advocate the increased use of simulation-based training for this anesthetic. The authors developed a scoring system to measure resident performance of this anesthetic on the human patient simulator and subjected the system to tests of validity and reliability. Methods:A modified Delphi technique was used to achieve a consensus among several experts regarding a standardized scoring system for evaluating resident performance of general anesthesia for emergency cesarean delivery on the human patient simulator. Eight third-year and eight first-year anesthesiology residents performed the scenario and were videotaped and scored by four attending obstetric anesthesiologists. Results:Third-year residents scored an average of 150.5 points, whereas first-year residents scored an average of 128 points (P = 0.004). The scoring instrument demonstrated high interrater reliability with an intraclass correlation coefficient of 0.97 (95% confidence interval, 0.94–0.99) compared with the average score. Conclusions:The developed scoring tool to measure resident performance of general anesthesia for emergency cesarean delivery on the patient simulator seems both valid and reliable in the context in which it was tested. This scoring system may prove useful for future studies such as those investigating the effect of simulator training on objective assessment of resident performance.

[1]  F C Forrest,et al.  Use of a high-fidelity simulator to develop testing of the technical performance of novice anaesthetists. , 2002, British journal of anaesthesia.

[2]  D. Gaba,et al.  Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings , 1998, Anesthesiology.

[3]  M. Maze,et al.  Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. , 2003, British journal of anaesthesia.

[4]  C. Gevirtz,et al.  Anesthesia-related deaths during obstetric delivery in the United States. 1979-1990. , 1997, Anesthesiology.

[5]  Anne K Wong,et al.  Full scale computer simulators in anesthesia training and evaluation , 2004, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[6]  J. D. Greaves,et al.  Assessment instruments used during anaesthetic simulation: review of published studies. , 2001, British journal of anaesthesia.

[7]  Morgan,et al.  Evaluation of medical students’ performance using the anaesthesia simulator , 2000, Medical education.

[8]  Joy L. Hawkins,et al.  Anesthesia-Related Deaths During Obstetric Delivery in the United States, 1979-1990 , 1997 .

[9]  Denise Polit-O'Hara,et al.  Nursing Research: Principles and Methods , 1978 .

[10]  Mark Clayton.  Delphi: a technique to harness expert opinion for critical decision‐making tasks in education , 1997 .

[11]  M. Cohen,et al.  The Validity of Performance Assessments Using Simulation , 2001, Anesthesiology.

[12]  Jodi Herold,et al.  Validity and reliability of undergraduate performance assessments in an anesthesia simulator , 2001, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[13]  David Murray,et al.  Reliability and Validity of a Simulation-based Acute Care Skills Assessment for Medical Students and Residents , 2003, Anesthesiology.

[14]  J. Woodhouse,et al.  Acute Care Skills in Anesthesia Practice: A Simulation-based Resident Performance Assessment , 2004, Anesthesiology.

[15]  Steve Lipman,et al.  The demise of general anesthesia in obstetrics revisited: prescription for a cure. , 2005, International journal of obstetric anesthesia.

[16]  H. Schwid,et al.  Evaluation of Anesthesia Residents Using Mannequin-based Simulation: A Multiinstitutional Study , 2002, Anesthesiology.

[17]  James R. Anderson,et al.  Obstetric anesthesia workforce survey: twenty-year update. , 2005, Anesthesiology.