Evaluation of a Measurement System to Assess ICU Team Performance*

Objective: Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs. Design: Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects. Setting: A 15-bed surgical ICU at a large academic hospital. Subjects: One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises. Interventions: No intervention was conducted for this study. Measurements and Main Results: Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64–0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity. Conclusions: Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

[1]  Sallie J. Weaver,et al.  Teamwork in Healthcare: Key Discoveries Enabling Safer, High-Quality Care , 2018, The American psychologist.

[2]  Interprofessional team interactions about complex care in the ICU: pilot development of an observational rating tool , 2016, BMC Research Notes.

[3]  Shannon L. Marlow,et al.  Saving lives: A meta-analysis of team training in healthcare. , 2016, The Journal of applied psychology.

[4]  E. Salas,et al.  Development of a Behavioral Marker System to Assess Intensive Care Unit Team Performance , 2015 .

[5]  C. V. D. van der Vleuten,et al.  The Reliability of Multisource Feedback in Competency-Based Assessment Programs: The Effects of Multiple Occasions and Assessor Groups , 2015, Academic medicine : journal of the Association of American Medical Colleges.

[6]  P. Pronovost,et al.  A systematic review of teamwork in the intensive care unit: what do we know about teamwork, team tasks, and improvement strategies? , 2014, Journal of critical care.

[7]  P. Pronovost,et al.  A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application? , 2014, BMJ quality & safety.

[8]  Paul J Sharek,et al.  Burnout in the NICU setting and its relation to safety culture , 2014, BMJ quality & safety.

[9]  Sallie J. Weaver,et al.  Team-training in healthcare: a narrative synthesis of the literature , 2014, BMJ quality & safety.

[10]  T. Manser,et al.  Do team processes really have an effect on clinical performance? A systematic literature review. , 2013, British journal of anaesthesia.

[11]  Kevin J O'Leary,et al.  Assessment of teamwork during structured interdisciplinary rounds on medical units. , 2012, Journal of hospital medicine.

[12]  Moshe Feldman,et al.  Rater training to support high-stakes simulation-based assessments. , 2012, The Journal of continuing education in the health professions.

[13]  Nick Sevdalis,et al.  Observational Teamwork Assessment for Surgery: Feasibility of Clinical and NonClinical Assessor Calibration With Short-Term Training , 2012, Annals of surgery.

[14]  Rhona Flin,et al.  Evaluation of the Scrub Practitioners' List of Intraoperative Non-Technical Skills system. , 2012, International journal of nursing studies.

[15]  Peter J Pronovost,et al.  Reducing medical errors and adverse events. , 2012, Annual review of medicine.

[16]  J. Beard,et al.  Prospective observational study to evaluate NOTSS (Non‐Technical Skills for Surgeons) for assessing trainees' non‐technical performance in the operating theatre , 2011, The British journal of surgery.

[17]  Boaz Shulruf,et al.  Evaluation of an instrument to measure teamwork in multidisciplinary critical care teams , 2011, Quality and Safety in Health Care.

[18]  J. Neily,et al.  Association between implementation of a medical team training program and surgical mortality. , 2010, JAMA.

[19]  Nick Sevdalis,et al.  Observational Teamwork Assessment for Surgery: Construct Validation With Expert Versus Novice Raters , 2009, Annals of surgery.

[20]  R. Flin,et al.  Developing a team performance framework for the intensive care unit* , 2009, Critical care medicine.

[21]  Eduardo Salas,et al.  Does Team Training Improve Team Performance? A Meta-Analysis , 2008, Hum. Factors.

[22]  Ronald F. Piccolo,et al.  A META‐ANALYSIS OF TEAMWORK PROCESSES: TESTS OF A MULTIDIMENSIONAL MODEL AND RELATIONSHIPS WITH TEAM EFFECTIVENESS CRITERIA , 2008 .

[23]  C. Roberts,et al.  ‘I’m pickin' up good regressions': the governance of generalisability analyses , 2007, Medical education.

[24]  Peter J Pronovost,et al.  Toward learning from patient safety reporting systems. , 2006, Journal of critical care.

[25]  R. Flin,et al.  Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. , 2003, British journal of anaesthesia.

[26]  John E. Mathieu,et al.  A Temporally Based Framework and Taxonomy of Team Processes , 2001 .

[27]  Lynne Martin,et al.  Behavioral Markers for Crew Resource Management: A Review of Current Practice , 2001 .

[28]  Catherine E. Volpe,et al.  Defining Competencies and Establishing Team Training Requirements , 1995 .

[29]  D. Cicchetti.  Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. , 1994 .

[30]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[31]  K. Kraiger,et al.  Generalizability Theory as Construct-Related Evidence of the Validity of Job Performance Ratings , 1990 .

[32]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[33]  Linda Allal,et al.  THE SYMMETRY OF GENERALIZABILITY THEORY: APPLICATIONS TO EDUCATIONAL MEASUREMENT , 1976 .

[34]  D. Weiss,et al.  Interrater reliability and agreement of subjective judgments , 1975 .