Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery

PurposeWith the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge.MethodsWe propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels.ResultsWe perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1–3 second window, without needing an observation of entire training trial.ConclusionThis study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

[1]  Sotirios A. Tsaftaris,et al.  Medical Image Computing and Computer Assisted Intervention , 2017 .

[2]  M. Bowyer,et al.  Modern Metrics for Evaluating Surgical Technical Skills , 2017, Current Surgery Reports.

[3]  Amod Jog,et al.  Assessing system operation skills in robotic surgery trainees , 2012, The international journal of medical robotics + computer assisted surgery : MRCAS.

[4]  Ana Luisa Trejos,et al.  Development of force-based metrics for skills assessment in minimally invasive surgery , 2014, Surgical Endoscopy.

[5]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[6]  Germain Forestier,et al.  Discovering Discriminative and Interpretable Patterns for Surgical Motion Analysis , 2017, AIME.

[7]  Marzieh Ershad,et al.  Meaningful Assessment of Surgical Expertise: Semantic Labeling with Data and Crowds , 2016, MICCAI.

[8]  Irfan A. Essa,et al.  Automated surgical skill assessment in RMIS training , 2017, International Journal of Computer Assisted Radiology and Surgery.

[9]  Ana Luisa Trejos,et al.  Analysis of Energy-Based Metrics for Laparoscopic Skills Assessment , 2018, IEEE Transactions on Biomedical Engineering.

[10]  Gregory D. Hager,et al.  Recognizing Surgical Activities with Recurrent Neural Networks , 2016, MICCAI.

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Gregory D. Hager,et al.  A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery , 2017, IEEE Transactions on Biomedical Engineering.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  John Cristian Borges Gamboa,et al.  Deep Learning for Time-Series Analysis , 2017, ArXiv.

[15]  A. Goh,et al.  Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. , 2012, The Journal of urology.

[16]  Xiaodong Gu,et al.  Max-Pooling Dropout for Regularization of Convolutional Neural Networks , 2015, ICONIP.

[17]  Narges Ahmidi,et al.  Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment. , 2016, Journal of surgical education.

[18]  Geoffrey I. Webb,et al.  Encyclopedia of Machine Learning , 2011, Encyclopedia of Machine Learning.

[19]  Yixin Chen,et al.  Multi-Scale Convolutional Neural Networks for Time Series Classification , 2016, ArXiv.

[20]  Satish T. S. Bukkapatnam,et al.  Time series forecasting for nonlinear and non-stationary processes: a review and comparative study , 2015 .

[21]  Inderbir S. Gill,et al.  External validation of Global Evaluative Assessment of Robotic Skills (GEARS) , 2015, Surgical Endoscopy.

[22]  Yuan Xing,et al.  Motion control skill assessment based on kinematic analysis of robotic end‐effector movements , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[23]  M. Iskander,et al.  The assessment of surgical competence , 2015, Medical teacher.

[24]  R. Reznick,et al.  Teaching surgical skills--changes in the wind. , 2006, The New England journal of medicine.

[25]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[26]  Dana Kulic,et al.  Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks , 2017, ICMI.

[27]  Michael D Klein,et al.  Automated robot‐assisted surgical skill evaluation: Predictive analytics approach , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[28]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[29]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[32]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[33]  Ilana Nisky,et al.  Instrument Orientation-Based Metrics for Surgical Skill Evaluation in Robot-Assisted and Open Needle Driving , 2017, ArXiv.

[34]  Katherine J. Kuchenbecker,et al.  Using Contact Forces and Robot Arm Accelerations to Automatically Rate Surgeon Skill at Peg Transfer , 2017, IEEE Trans. Biomed. Eng..

[35]  Henry C. Lin,et al.  JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ) : A Surgical Activity Dataset for Human Motion Modeling , 2014 .

[36]  K. M. Deliparaschos,et al.  Evolution of autonomous and semi‐autonomous robotic surgical systems: a review of the literature , 2011, The international journal of medical robotics + computer assisted surgery : MRCAS.

[37]  Gregory D. Hager,et al.  Sparse Hidden Markov Models for Surgical Gesture Classification and Skill Evaluation , 2012, IPCAI.

[38]  W. Klonowski.  Everything you wanted to ask about EEG but were afraid to get the right answer , 2009, Nonlinear biomedical physics.

[39]  J. Dankelman,et al.  Objective classification of residents based on their psychomotor laparoscopic skills , 2009, Surgical Endoscopy.

[40]  Amy Loutfi,et al.  A review of unsupervised feature learning and deep learning for time-series modeling , 2014, Pattern Recognit. Lett..

[41]  Jones Mark,et al.  Surgeon specific mortality in adult cardiac surgery: comparison between crude and risk stratified data , 2003, BMJ : British Medical Journal.

[42]  Rajnikant V. Patel,et al.  Energy-Based Metrics for Arthroscopic Skills Assessment , 2017, Sensors.

[43]  Masaru Ishii,et al.  Objective Assessment of Surgical Technical Skill and Competency in the Operating Room. , 2017, Annual review of biomedical engineering.

[44]  R. Reznick,et al.  Training and simulation for patient safety , 2010, Quality and Safety in Health Care.

[45]  A. Duffy,et al.  Evolution of surgical skills training. , 2006, World journal of gastroenterology.

[46]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[47]  Gregory D. Hager,et al.  Task versus Subtask Surgical Skill Evaluation of Robotic Minimally Invasive Surgery , 2009, MICCAI.

[48]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[49]  J. Birkmeyer,et al.  Surgical skill and complication rates after bariatric surgery. , 2013, The New England journal of medicine.

[50]  Alexander J. Smola,et al.  Efficient mini-batch training for stochastic optimization , 2014, KDD.

[51]  Andrew Y. Ng,et al.  Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks , 2017, ArXiv.

[52]  Danail Stoyanov,et al.  Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions , 2016, International Journal of Computer Assisted Radiology and Surgery.

[53]  Naoki Hirabayashi,et al.  Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room , 2012, Surgery Today.

[54]  Henry C. Lin,et al.  Review of methods for objective surgical skill evaluation , 2011, Surgical Endoscopy.

[55]  T. Judkins,et al.  Objective evaluation of expert and novice performance during robotic surgical training tasks , 2009, Surgical Endoscopy.

[56]  James R Korndorffer,et al.  Do Metrics Matter? Time Versus Motion Tracking for Performance Assessment of Proficiency-Based Laparoscopic Skills Training , 2009, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[57]  Timothy M. Kowalewski,et al.  The minimally acceptable classification criterion for surgical skill: intent vectors and separability of raw motion data , 2017, International Journal of Computer Assisted Radiology and Surgery.

[58]  Romain Tavenard,et al.  Data Augmentation for Time Series Classification using Convolutional Neural Networks , 2016 .