Observational Teamwork Assessment for Surgery: Construct Validation With Expert Versus Novice Raters

Objective:To test the construct validity of the Observational Teamwork Assessment for Surgery (OTAS) tool. Summary Background Data:Poor teamwork in surgical teams has been implicated in adverse events to patients. The OTAS is a tool that assesses teamwork in real time for the entire surgical team. Existing empirical research on OTAS has yet to explore how expert versus novice tool users use the tool to assess teamwork in the operating room. Methods:Data were collected in 12 elective procedures by an expert/expert (N = 6) and an expert/novice (N = 6) pair of raters. Five teamwork behaviors (communication, coordination, leadership, monitoring, and cooperation) were scored via observation pre, intra, and postoperatively by blind raters. Results:Significant and sizeable correlations were obtained in 12 of 15 behaviors in the expert/expert pair, but only in 3 of 15 behaviors in the expert/novice pair. Significant differences in mean scores were obtained in 3 of 15 behaviors in the expert/expert pair, but in 11 of 15 behaviors in the expert/novice pair. Total OTAS scores exhibited strong correlations and no significant differences in ratings in the expert/expert pair. In the expert/novice pair no correlations were obtained and there were significant differences in mean scores. The overall size of inconsistency in the scoring was 2% for expert/expert versus 15% for expert/novice. Conclusions:OTAS exhibits adequate construct validity as assessed by consistency in the scoring by expert versus novices—ie, expert raters produce significantly more consistent scoring than novice raters. Further validation should assess the learning curve for novices in OTAS. Relationships between OTAS, measures of technical skill, and behavioral responses to surgical crises should also be quantified.

[1]  Nick Sevdalis,et al.  Mapping surgical practice decision making: an interview study to evaluate decisions in surgical care. , 2008, American journal of surgery.

[2]  R. Flin,et al.  Non-technical skills for surgeons in the operating room: a review of the literature. , 2006, Surgery.

[3]  P. McCulloch,et al.  Teamwork and Error in the Operating Room: Analysis of Skills and Roles , 2008, Annals of surgery.

[4]  Douglas A Wiegmann,et al.  Disruptions in surgical flow and their relationship to surgical errors: an exploratory investigation. , 2007, Surgery.

[5]  Nick Sevdalis,et al.  Annoyances, Disruptions, and Interruptions in Surgery: The Disruptions in Surgery Index (DiSI) , 2008, World Journal of Surgery.

[6]  Gerald M. Fried,et al.  Objective Assessment of Technical Performance , 2008, World Journal of Surgery.

[7]  J Annett,et al.  A method for measuring team skills , 2000, Ergonomics.

[8]  C R Paris,et al.  Teamwork in multi-person systems: a review and analysis , 2000, Ergonomics.

[9]  S Undre,et al.  Developing observational measures of performance in surgical teams , 2004, Quality and Safety in Health Care.

[10]  Nick Sevdalis,et al.  Judgment analysis: a method for quantitative evaluation of trainee surgeons' judgments of surgical risk. , 2008, American journal of surgery.

[11]  Robert M Mentzer,et al.  Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. , 2007, Journal of the American College of Surgeons.

[12]  J Carthey,et al.  The role of structured observational research in health care , 2003, Quality & safety in health care.

[13]  Lorelei Lingard,et al.  Team Communications in the Operating Room: Talk Patterns, Sites of Tension, and Implications for Novices , 2002, Academic medicine : journal of the Association of American Medical Colleges.

[14]  Ara Darzi,et al.  Observational Assessment of Surgical Teamwork: A Feasibility Study , 2006, World Journal of Surgery.

[15]  Stuart R Lipsitz,et al.  Patterns of communication breakdowns resulting in injury to surgical patients. , 2007, Journal of the American College of Surgeons.

[16]  Nick Sevdalis,et al.  Multidisciplinary Crisis Simulations: The Way Forward for Training Surgical Teams , 2007, World Journal of Surgery.

[17]  S. Urbina,et al.  Psychological testing, 7th ed. , 1997 .

[18]  Rajesh Aggarwal,et al.  A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. , 2006, American journal of surgery.

[19]  Stephanie Guerlain,et al.  Assessing team performance in the operating room: development and use of a "black-box" recorder and other tools for the intraoperative environment. , 2005, Journal of the American College of Surgeons.

[20]  David M Studdert,et al.  Analysis of errors reported by surgeons at three teaching hospitals. , 2003, Surgery.

[21]  Nick Sevdalis,et al.  Managing intraoperative stress: what do surgeons want from a crisis training program? , 2009, American journal of surgery.

[22]  Charles Vincent,et al.  Systems Approaches to Surgical Quality and Safety: From Concept to Measurement , 2004, Annals of surgery.

[23]  Ara Darzi,et al.  Surgical Crisis Management Skills Training and Assessment: A Stimulation-Based Approach to Enhancing Operating Room Performance , 2006 .

[24]  R. Reznick,et al.  Communication failures in the operating room: an observational classification of recurrent types and effects , 2004, Quality and Safety in Health Care.

[25]  Rajesh Aggarwal,et al.  An Evidence-Based Virtual Reality Training Program for Novice Laparoscopic Surgeons , 2006, Annals of surgery.

[26]  C. Vincent,et al.  Distracting communications in the operating theatre. , 2007, Journal of evaluation in clinical practice.

[27]  R. Mcintyre,et al.  A Conceptual Framework for Teamwork Measurement , 1997 .

[28]  M R de Leval,et al.  The human factor in cardiac surgery: errors and near misses in a high technology medical domain. , 2001, The Annals of thoracic surgery.

[29]  T. Brennan,et al.  INCIDENCE OF ADVERSE EVENTS AND NEGLIGENCE IN HOSPITALIZED PATIENTS , 2008 .

[30]  Nick Sevdalis,et al.  Observational Teamwork Assessment for Surgery (OTAS): Refinement and Application in Urological Surgery , 2007, World Journal of Surgery.

[31]  Nick Sevdalis,et al.  The complexity of measuring interprofessional teamwork in the operating theatre , 2006, Journal of interprofessional care.

[32]  N. Sevdalis,et al.  Measuring intra-operative interference from distraction and interruption observedin the operating theatre , 2006, Ergonomics.

[33]  M Koutantji,et al.  Quantifying distraction and interruption in urological surgery , 2007, Quality and Safety in Health Care.

[34]  S. Guerlain,et al.  A systems approach to surgical safety , 2002, Surgical Endoscopy And Other Interventional Techniques.

[35]  S Undre,et al.  Defining the technical skills of teamwork in surgery , 2006, Quality and Safety in Health Care.

[36]  P. McCulloch,et al.  The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy , 2007, Surgical Endoscopy.

[37]  Jane Kidd,et al.  The effects of stress on surgical performance. , 2006, American journal of surgery.

[38]  Nick Sevdalis,et al.  Teamwork in the operating theatre: cohesion or confusion? , 2006, Journal of evaluation in clinical practice.

[39]  Julia Neily,et al.  Teamwork and communication in surgical teams: implications for patient safety. , 2008, Journal of the American College of Surgeons.

[40]  R. Reznick,et al.  Evaluation of a preoperative checklist and team briefing among surgeons, nurses, and anesthesiologists to reduce failures in communication. , 2008, Archives of surgery.

[41]  Stuart R. Lipsitz,et al.  Patterns of Communication Breakdowns Resulting in Injury to Surgical Patients , 2007 .