Observational clinical human reliability analysis (OCHRA) for competency assessment in laparoscopic colorectal surgery at the specialist level

BackgroundThere are no valid and reliable tools to assess competency in advanced laparoscopic surgery at a specialist level. The observational clinical human reliability analysis (OCHRA) may have the required characteristics of such a tool. The aim of this study was to evaluate construct and concurrent validity of OCHRA for competency assessment at a specialist level.MethodsThirty-two video-recorded laparoscopic colorectal resections, performed by experts and delegates of the National Training Program in England, were evaluated. Each video was analysed using OCHRA by identifying errors enacted during surgery. The number of tissue-handling, instrument-misuse, and consequential errors was recorded using video-rating software. Times spent on dissecting (D) and on exposing (E) tissues were also measured (D/E ratio). In addition, two independent expert surgeons globally assessed each video regarding competency (pass vs. fail). Logistic regression was used to predict outcomes.ResultsA total of 399 errors were identified. There was a significant difference when comparing the expert, pass, and fail groups for total errors (median counts for experts = 4, pass = 10, fail = 17; P < 0.001). When comparing the pass and fail groups excluding experts, differences could be found for tissue-handling errors (7 vs. 12; P = 0.005), but not for consequential errors (4 vs. 7; P = 0.059) and instrument-handling errors (4 vs. 5; P = 0.320). The D/E ratio was significantly lower for delegates than for experts (0.6 vs. 1.0; P = 0.001). When all four independent variables were used to predict delegates who passed or failed, the area under the receiver operating characteristic curve was 0.867, sensitivity was 71.4%, and specificity was 90.9%.ConclusionOCHRA is a valid tool for assessing competency at a specialist level in advanced laparoscopic surgery. It has the potential to be used for recertification and revalidation of specialists.

[1]  B Kirwan,et al.  Human error identification in human reliability assessment. Part 1: Overview of approaches. , 1992, Applied ergonomics.

[2]  J. Concato,et al.  Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. , 1995, Journal of clinical epidemiology.

[3]  R. Reznick,et al.  Objective structured assessment of technical skill (OSATS) for surgical residents , 1997, The British journal of surgery.

[4]  A. Cuschieri,et al.  Computer-controlled endoscopic performance assessment system , 1998, Surgical Endoscopy.

[5]  A. Cuschieri,et al.  Errors enacted during endoscopic surgery--a human reliability analysis. , 1998, Applied ergonomics.

[6]  Alfred Cuschieri,et al.  The performance of master surgeons on the Advanced Dundee Endoscopic Psychomotor Tester: contrast validity study. , 2002, Archives of surgery.

[7]  G. Fried,et al.  Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. , 2003, Surgical endoscopy.

[8]  A. Darzi,et al.  Motion analysis in the training and assessment of minimally invasive surgery , 2003, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[9]  R. M. Satava,et al.  Discriminative validity of the Minimally Invasive Surgical Trainer in Virtual Reality (MIST-VR) using criteria levels based on expert performance , 2004, Surgical Endoscopy And Other Interventional Techniques.

[10]  Krishna Moorthy,et al.  Ceiling effect in technical skills of surgical residents. , 2004, American journal of surgery.

[11]  A Cuschieri,et al.  Identification and categorization of technical errors by Observational Clinical Human Reliability Assessment (OCHRA) during laparoscopic cholecystectomy. , 2004, Archives of surgery.

[12]  A. Cuschieri,et al.  Analysis of technical surgical errors during initial experience of laparoscopic pyloromyotomy by a group of dutch pediatric surgeons , 2004, Surgical Endoscopy.

[13]  A Cuschieri,et al.  Analysis of errors enacted by surgical trainees during skills training courses. , 2005, Surgery.

[14]  C. V. D. van der Vleuten,et al.  Assessing professional competence : from methods to programmes , 2005 .

[15]  C. Delaney,et al.  Evaluation of the Learning Curve in Laparoscopic Colorectal Surgery: Comparison of Right-Sided and Left-Sided Resections , 2005, Annals of surgery.

[16]  G. Fried,et al.  The MISTELS program to measure technical skill in laparoscopic surgery , 2006, Surgical Endoscopy And Other Interventional Techniques.

[17]  Melina C Vassiliou,et al.  A global assessment tool for evaluation of intraoperative laparoscopic skills. , 2005, American journal of surgery.

[18]  A. Darzi,et al.  Technical skills examination for general surgical trainees , 2006, The British journal of surgery.

[19]  Rajesh Aggarwal,et al.  A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. , 2006, American journal of surgery.

[20]  Rajesh Aggarwal,et al.  Toward Feasible, Valid, and Reliable Video-Based Assessments of Technical Surgical Skills in the Operating Room , 2008, Annals of surgery.

[21]  E. Sabri,et al.  Laparoscopic Colon Surgery: Does Operative Time Matter? , 2009, Diseases of the colon and rectum.

[22]  E. Verdaasdonk,et al.  Objective assessment of technical surgical skills , 2010, The British journal of surgery.

[23]  K. Mizokami,et al.  Accelerated learning curve for colorectal resection, open versus laparoscopic approach, can be attained with expert supervision , 2010, Surgical Endoscopy.

[24]  Gerald M. Fried,et al.  Trends and results of the first 5 years of Fundamentals of Laparoscopic Surgery (FLS) certification testing , 2010, Surgical Endoscopy.

[25]  A. Darzi,et al.  Systematic Review on Mentoring and Simulation in Laparoscopic Colorectal Surgery , 2010, Annals of surgery.

[26]  G. Hanna,et al.  Development, validation and implementation of a monitoring tool for training in laparoscopic colorectal surgery in the English National Training Program , 2011, Surgical Endoscopy.

[27]  G. Hanna,et al.  The National Training Programme for Laparoscopic Colorectal Surgery in England: a new training paradigm , 2011, Colorectal disease : the official journal of the Association of Coloproctology of Great Britain and Ireland.

[28]  A. Darzi,et al.  Observational tools for assessment of procedural skills: a systematic review. , 2011, American journal of surgery.

[29]  G. Hanna,et al.  Root cause analysis of internal hernia and Roux limb compression after laparoscopic Roux-en-Y gastric bypass using observational clinical human reliability assessment. , 2012, Surgery for obesity and related diseases : official journal of the American Society for Bariatric Surgery.