Development and validation of a tool to assess emergency physicians' nontechnical skills.

STUDY OBJECTIVE Nontechnical skills are "the cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance." Our research team developed and evaluated the task of developing and validating a behavioral marker system for the observational assessment of emergency physicians' nontechnical skills. METHODS The development of the tool was divided into 3 phases and used triangulation of data from a number of sources. During phase 1, a provisional assessment tool was developed according to published literature and curricula. Phase 2 used analysis of staff interviews and field observations to determine whether the skill list contained any significant omissions. These studies were also used to identify behavioral markers linked to nontechnical skills in the context of the emergency department (ED) and establish whether skills included in the tool were observable. Phase 3 involved evaluating the content validity index of exemplar behaviors, using a survey of experts. RESULTS A behavioral marker system was developed that comprised 12 emergency medicine-specific nontechnical skills, grouped into 4 categories. Content validity was assessed with a survey of 148 emergency medicine staff, and 75% of items achieved the recommended content validity index greater than 0.75. Data from the survey enabled further refinement of the behavioral markers to produce a final version of the tool. CONCLUSION Although further evaluative studies are needed, this behavioral marker system provides a structured approach to the assessment and training of nontechnical skills in the ED.

[1]  Catherine Yoon,et al.  Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. , 2007, Annals of emergency medicine.

[2]  Douglas A Wiegmann,et al.  The Mayo High Performance Teamwork Scale: Reliability and Validity for Evaluating Key Crew Resource Management Skills , 2007, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[3]  Anne Lippert,et al.  Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. , 2010, Resuscitation.

[4]  D. Cook,et al.  Does scale length matter? A comparison of nine- versus five-point rating scales for the mini-CEX , 2009, Advances in health sciences education : theory and practice.

[5]  R. Flin,et al.  Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. , 2003, British journal of anaesthesia.

[6]  J. Sexton,et al.  Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation , 2004, Quality and Safety in Health Care.

[7]  M. Lynn.  Determination and quantification of content validity. , 1986, Nursing research.

[8]  James L Smythe,et al.  Critical incident monitoring is a useful quality improvement tool for the emergency department , 2009 .

[9]  M F Ben-David,et al.  The role of assessment in expanding professional horizons , 2000, Medical teacher.

[10]  J. Norcini,et al.  Work based assessment , 2003, BMJ : British Medical Journal.

[11]  Debra Nestel,et al.  Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). , 2010, Resuscitation.

[12]  Akihito Kamata,et al.  Developing and Validating Rapid Assessment Instruments , 2009 .

[13]  R. Roberts,et al.  Characteristics of patient care management problems identified in emergency department morbidity and mortality investigations during 15 years. , 2008, Annals of emergency medicine.

[14]  Work based assessment ABC of learning and teaching in medicine , 2005 .

[15]  C. Vincent,et al.  Teams under pressure in the emergency department: an interview study , 2011, Emergency Medicine Journal.

[16]  Amer A. Kaissi,et al.  Measuring teamwork and patient safety attitudes of high-risk areas. , 2003, Nursing economic$.

[17]  John C. Morey,et al.  Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. , 2002, Health services research.

[18]  Nick Sevdalis,et al.  Observational teamwork assessment for surgery: content validation and tool refinement. , 2011, Journal of the American College of Surgeons.

[19]  S D Berns,et al.  The potential for improved teamwork to reduce medical errors in the emergency department. The MedTeams Research Consortium. , 1999, Annals of emergency medicine.

[20]  C. Pope,et al.  Teams, talk and transitions in anaesthetic practice , 2009 .

[21]  L. Davis,et al.  Selection and use of content experts for instrument development. , 1997, Research in nursing & health.

[22]  K. Eva,et al.  The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom , 2009, Advances in health sciences education : theory and practice.

[23]  S Undre,et al.  Developing observational measures of performance in surgical teams , 2004, Quality and Safety in Health Care.

[24]  Michelle Chiu,et al.  A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study , 2006, Critical care medicine.

[25]  Eduardo Salas,et al.  Developing Teams and Team Leaders: Strategies and Principles. , 2004 .

[26]  Rhona Flin,et al.  Safety at the Sharp End: A Guide to Non-Technical Skills , 2008 .

[27]  Rhona Flin,et al.  Surgeons’ Non-technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System , 2008, World Journal of Surgery.

[28]  Kenji Hira,et al.  Evaluation of trauma team performance using an advanced human patient simulator for resuscitation training. , 2002, The Journal of trauma.

[29]  C. Vincent,et al.  Identifying nontechnical skills associated with safety in the emergency department: a scoping review of the literature. , 2012, Annals of emergency medicine.

[30]  V. Martuza.  Applying norm-referenced and criterion-referenced measurement in education , 1977 .

[31]  Katherine A. Wilson,et al.  Crew Resource Management Training Effectiveness: A Meta-Analysis and Some Critical Needs , 2008 .

[32]  D. Irby,et al.  Assessment in medical education. , 2007, The New England journal of medicine.

[33]  R Flin,et al.  Development of a rating system for surgeons' non‐technical skills , 2006, Medical education.

[34]  Michael Fordis,et al.  Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. , 2006, JAMA.

[35]  John Whittington,et al.  SBAR: a shared mental model for improving communication between clinicians. , 2006, Joint Commission journal on quality and patient safety.

[36]  D. Kilroy.  Clinical supervision in the emergency department: a critical incident study , 2006, Emergency Medicine Journal.

[37]  Peter Oluf Andersen,et al.  Critical incidents related to cardiac arrests reported to the Danish Patient Safety Database. , 2010, Resuscitation.

[38]  G. Lighthall,et al.  Evaluating the management of septic shock using patient simulation , 2007, Critical care medicine.

[39]  Roxane Gardner,et al.  Using the Communication and Teamwork Skills (CATS) Assessment to measure health care team performance. , 2007, Joint Commission journal on quality and patient safety.

[40]  K. Sutcliffe,et al.  Resident perceptions of medical errors in the emergency department. , 2003, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[41]  G. Setnik,et al.  Demonstration of high-fidelity simulation team training for emergency medicine. , 1999, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[42]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[43]  G. Jenkins,et al.  A Monte Carlo study of factors affecting three indices of composite scale reliability. , 1977 .

[44]  Cheryl Tatano Beck,et al.  Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. , 2007, Research in nursing & health.

[45]  P. McCulloch,et al.  The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre , 2009, Quality & Safety in Health Care.

[46]  K. Cosby,et al.  A framework for classifying factors that contribute to error in the emergency department. , 2003, Annals of emergency medicine.

[47]  Cindy Irwin,et al.  Cause-and-effect analysis of risk management files to assess patient care in the emergency department. , 2004, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[48]  T. Manser.  Teamwork and patient safety in dynamic domains of healthcare: a review of the literature , 2009, Acta anaesthesiologica Scandinavica.

[49]  E. Salas,et al.  Toward a definition of teamwork in emergency medicine. , 2008, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[50]  A. Wragg,et al.  Implementing workplace‐based assessment across the medical specialties in the United Kingdom , 2008, Medical education.