Automated Near–Real-time Clinical Performance Feedback for Anesthesiology Residents: One Piece of the Milestones Puzzle

Background:Anesthesiology residencies are developing trainee assessment tools to evaluate 25 milestones that map to the six core competencies. The effort will be facilitated by development of automated methods to capture, assess, and report trainee performance to program directors, the Accreditation Council for Graduate Medical Education and the trainees themselves. Methods:The authors leveraged a perioperative information management system to develop an automated, near–real-time performance capture and feedback tool that provides objective data on clinical performance and requires minimal administrative effort. Before development, the authors surveyed trainees about satisfaction with clinical performance feedback and about preferences for future feedback. Results:Resident performance on 24,154 completed cases has been incorporated into the authors’ automated dashboard, and trainees now have access to their own performance data. Eighty percent (48 of 60) of the residents responded to the feedback survey. Overall, residents “agreed/strongly agreed” that they desire frequent updates on their clinical performance on defined quality metrics and that they desired to see how they compared with the residency as a whole. Before deployment of the new tool, they “disagreed” that they were receiving feedback in a timely manner. Survey results were used to guide the format of the feedback tool that has been implemented. Conclusion:The authors demonstrate the implementation of a system that provides near–real-time feedback concerning resident performance on an extensible series of quality metrics, and which is responsive to requests arising from resident feedback about desired reporting mechanisms.

[1]  Eric S. Holmboe,et al.  Promoting physicians' self‐assessment and quality improvement: The ABIM diabetes practice improvement module , 2006, The Journal of continuing education in the health professions.

[2]  Using an anesthesia information management system to improve case log data entry and resident workflow. , 2011, Anesthesia and analgesia.

[3]  Jon Veloski,et al.  Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7 , 2006, Medical teacher.

[4]  Warren S Sandberg,et al.  Using information technology to improve quality in the OR. , 2011, Anesthesiology clinics.

[5]  Perioperative normothermia to reduce the incidence of surgical-wound infection and shorten hospitalization. Study of Wound Infection and Temperature Group. , 1996, The New England journal of medicine.

[6]  Franklin Dexter,et al.  Implications of Event Entry Latency on Anesthesia Information Management Decision Support Systems , 2009, Anesthesia and analgesia.

[7]  M. Hollmann,et al.  Decision Support Increases Guideline Adherence for Prescribing Postoperative Nausea and Vomiting Prophylaxis , 2008, Anesthesia and analgesia.

[8]  Lawrence T. Post,et al.  American College of Surgeons , 2020, Definitions.

[9]  Lee A Fleisher,et al.  Practice Guidelines for Central Venous Access: A Report by the American Society of Anesthesiologists Task Force on Central Venous Access , 2012, Anesthesiology.

[10]  A HarrisPaul,et al.  Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support , 2009 .

[11]  Jesse M. Ehrenfeld,et al.  Ongoing professional performance evaluation (OPPE) using automatically captured electronic anesthesia data. , 2012, Joint Commission journal on quality and patient safety.

[12]  K. Tremper,et al.  Electronic Reminders Improve Procedure Documentation Compliance and Professional Fee Reimbursement , 2007, Anesthesia and analgesia.

[13]  I. Philibert,et al.  The next GME accreditation system--rationale and benefits. , 2012, The New England journal of medicine.

[14]  R L Menlove,et al.  The timing of prophylactic administration of antibiotics and the risk of surgical-wound infection. , 1992, The New England journal of medicine.

[15]  F. Chung,et al.  Practice Guidelines for Postanesthetic Care: An Updated Report by the American Society of Anesthesiologists Task Force on Postanesthetic Care , 2013, Anesthesiology.

[16]  Jesse M. Ehrenfeld,et al.  Real-Time Checking of Electronic Anesthesia Records for Documentation Errors and Automatically Text Messaging Clinicians Improves Quality of Documentation , 2008, Anesthesia and analgesia.

[17]  J. Kroin,et al.  Multimodal analgesia for controlling acute postoperative pain , 2009, Current opinion in anaesthesiology.

[18]  A. Wilcox,et al.  An automated procedure logging system improves resident documentation compliance. , 2011, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[19]  R. Epstein,et al.  The Design and Implementation of an Automated System for Logging Clinical Experiences Using an Anesthesia Information Management System , 2011, Anesthesia and analgesia.

[20]  William C McGaghie,et al.  Comparison of Two Standard-setting Methods for Advanced Cardiac Life Support Training , 2005, Academic medicine : journal of the Association of American Medical Colleges.

[21]  John R Boulet,et al.  Setting defensible performance standards on OSCEs and standardized patient examinations , 2003, Medical teacher.

[22]  Keith Baker,et al.  Determining Resident Clinical Performance: Getting Beyond the Noise , 2011, Anesthesiology.

[23]  L. Lynn,et al.  Measuring Physicians’ Performance in Clinical Practice: Reliability, Classification Accuracy, and Validity , 2010, Evaluation & the health professions.

[24]  B. Hess,et al.  Physician performance assessment: prevention of cardiovascular disease , 2013, Advances in health sciences education : theory and practice.

[25]  Warren S. Sandberg,et al.  The incidence of hypoxemia during surgery: evidence from two institutions , 2010, Canadian journal of anaesthesia = Journal canadien d'anesthesie.

[26]  Anesthesia information management systems: almost there. , 2008, Anesthesia and analgesia.

[27]  Stephen F Spring,et al.  Automated Documentation Error Detection and Notification Improves Anesthesia Billing Performance , 2007, Anesthesiology.

[28]  AkkeNeel Talsma,et al.  An Anesthesia Information System Designed to Provide Physician-Specific Feedback Improves Timely Administration of Prophylactic Antibiotics , 2006, Anesthesia and analgesia.

[29]  Nassib G. Chamoun,et al.  Hospital Stay and Mortality Are Increased in Patients Having a “Triple Low” of Low Blood Pressure, Low Bispectral Index, and Low Minimum Alveolar Concentration of Volatile Anesthesia , 2012, Anesthesiology.

[30]  Jesse M. Ehrenfeld,et al.  Automatic Notifications Mediated by Anesthesia Information Management Systems Reduce the Frequency of Prolonged Gaps in Blood Pressure Documentation , 2011, Anesthesia and analgesia.

[31]  D. Bratzler,et al.  Antimicrobial prophylaxis for surgery: an advisory statement from the National Surgical Infection Prevention Project. , 2004, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[32]  E. Hessel,et al.  Anesthesiology resident personality type correlates with faculty assessment of resident performance. , 2012, Journal of clinical anesthesia.

[33]  Eric S. Holmboe,et al.  Setting a Fair Performance Standard for Physicians’ Quality of Patient Care , 2011, Journal of General Internal Medicine.

[34]  William C. McGaghie,et al.  Setting Defensible Standards for Cardiac Auscultation Skills in Medical Students , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[35]  D. Bratzler,et al.  Antimicrobial prophylaxis for surgery: an advisory statement from the National Surgical Infection Prevention Project. , 2004, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[36]  J. E. Anderson,et al.  Review of simulation studies in anaesthesia journals, 2001-2010: mapping and content analysis. , 2012, British journal of anaesthesia.

[37]  P. Harris,et al.  Research electronic data capture (REDCap) - A metadata-driven methodology and workflow process for providing translational research informatics support , 2009, J. Biomed. Informatics.