Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment.

OBJECTIVE Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment. DESIGN Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models. SETTING Robotic surgical skills training laboratory. PARTICIPANTS In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery. RESULTS Experts performed the task in shorter time (49.74s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17). CONCLUSIONS Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

[1]  Gregory D. Hager,et al.  Surgical Gesture Segmentation and Recognition , 2013, MICCAI.

[2]  A. Dosis,et al.  Motion tracking systems for assessment of surgical skill , 2007, Surgical Endoscopy.

[3]  Ara Darzi,et al.  Electromagnetic motion analysis in the assessment of surgical skill: Relationship between time and movement , 2002, ANZ journal of surgery.

[4]  Willi Sauerbrei,et al.  Bootstrap and Cross-Validation to Assess Complexity of Data-Driven Regression Models , 2000, ISMDA.

[5]  R. Reznick,et al.  Objective structured assessment of technical skill (OSATS) for surgical residents , 1997, The British journal of surgery.

[6]  Gregory D. Hager,et al.  Surgical gesture classification from video and kinematic data , 2013, Medical Image Anal..

[7]  M. Schemper.  Predictive accuracy and explained variation , 2003, Statistics in medicine.

[8]  A. Darzi,et al.  Measurement of Surgical Dexterity Using Motion Analysis of Simple Bench Tasks , 2003, World Journal of Surgery.

[9]  Amod Jog,et al.  Assessing system operation skills in robotic surgery trainees , 2012, The international journal of medical robotics + computer assisted surgery : MRCAS.

[10]  Lena Maier-Hein,et al.  Can Masses of Non-Experts Train Highly Accurate Image Classifiers? - A Crowdsourcing Approach to Instrument Segmentation in Laparoscopic Images , 2014, MICCAI.

[11]  Sanjeev Khudanpur,et al.  Learning and inference algorithms for dynamical system models of dextrous motion , 2011 .

[12]  Gregory D. Hager,et al.  Sparse Hidden Markov Models for Surgical Gesture Classification and Skill Evaluation , 2012, IPCAI.

[13]  D. Yuh,et al.  Objective measures for longitudinal assessment of robotic surgery training. , 2012, The Journal of thoracic and cardiovascular surgery.

[14]  Ara Darzi,et al.  The surgical error examination is a novel method for objective technical knowledge assessment. , 2003, American journal of surgery.

[15]  A. Darzi,et al.  The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. , 2001, Journal of the American College of Surgeons.

[16]  Henry C. Lin,et al.  Review of methods for objective surgical skill evaluation , 2011, Surgical Endoscopy.

[17]  Gregory D. Hager,et al.  String Motif-Based Description of Tool Motion for Detecting Skill and Gestures in Robotic Surgery , 2013, MICCAI.

[18]  Gregory D. Hager,et al.  Structure in surgical motion , 2010 .

[19]  Gregory D. Hager,et al.  A study of crowdsourced segment-level surgical skill assessment using pairwise rankings , 2015, International Journal of Computer Assisted Radiology and Surgery.

[20]  Antonino Freno,et al.  Selecting Features by Learning Markov Blankets , 2007, KES.

[21]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[22]  Marco Scutari,et al.  Learning Bayesian Networks with the bnlearn R Package , 2009, 0908.3817.

[23]  Ara Darzi,et al.  Surgical skill is predicted by the ability to detect errors. , 2005, American journal of surgery.

[24]  Jennifer L Pearson,et al.  Rapid Grading of Fundus Photographs for Diabetic Retinopathy Using Crowdsourcing , 2014, Journal of medical Internet research.

[25]  René Vidal,et al.  Surgical Gesture Classification from Video Data , 2012, MICCAI.

[26]  Gregory D. Hager,et al.  Data-Derived Models for Segmentation with Application to Surgical Assessment and Training , 2009, MICCAI.

[27]  W. Pan.  Akaike's Information Criterion in Generalized Estimating Equations , 2001, Biometrics.

[28]  Timothy M. Kowalewski,et al.  Crowd-Sourced Assessment of Technical Skills: a novel method to evaluate surgical performance. , 2014, The Journal of surgical research.

[29]  Ian J Dooley,et al.  Subjective difficulty of each stage of phacoemulsification cataract surgery performed by basic surgical trainees , 2006, Journal of cataract and refractive surgery.

[30]  Gregory D. Hager,et al.  Task versus Subtask Surgical Skill Evaluation of Robotic Minimally Invasive Surgery , 2009, MICCAI.