Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark

Martin Wagner MD, Beat-Peter Müller-Stich MD, Anna Kisilenko, Duc Tran, Patrick Heger MD, Lars Mündermann PhD, David M Lubotsky BSc, Benjamin Müller, Tornike Davitashvili, Manuela Capek, Annika Reinke, Tong Yu MSc, Armine Vardazaryan MSc, Chinedu Innocent Nwoye MSc, Nicolas Padoy PhD, Xinyang Liu, Eung-Joo Lee, Constantin Disch, Hans Meine PhD, Tong Xia BSc, Fucang Jia PhD, Satoshi Kondo PhD, Wolfgang Reiter, Yueming Jin, Yonghao Long, Meirui Jiang, Qi Dou, Pheng Ann Heng, Isabell Twick, Kadir Kirtac, Enes Hosgor, Jon Lindström Bolmgren, Michael Stenzel, Björn von Siemens, Hannes G. Kenngott MD MSc, Felix Nickel MD MME, Moritz von Frankenberg MD, Franziska Mathis-Ullrich PhD, Lena Maier-Hein PhD, Stefanie Speidel PhD*, Sebastian Bodenstedt PhD*

[1]  Tao Mei,et al.  Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[2]  Didier Mutter,et al.  Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos , 2018, International Journal of Computer Assisted Radiology and Surgery.

[3]  G. Hanna,et al.  Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery. , 2020, JAMA surgery.

[4]  Ronald M. Summers,et al.  ChestX-ray: Hospital-Scale Chest X-ray Database and Benchmarks on Weakly Supervised Classification and Localization of Common Thorax Diseases , 2019, Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics.

[5]  Peter M. Full,et al.  Heidelberg colorectal data set for surgical data science in the sensor operating room , 2020, Scientific Data.

[6]  Russell H. Taylor,et al.  Surgical data science for next-generation interventions , 2017, Nature Biomedical Engineering.

[7]  Pierre Jannin,et al.  Surgical process modelling: a review , 2014, International Journal of Computer Assisted Radiology and Surgery.

[8]  Masaru Ishii,et al.  Objective Assessment of Surgical Technical Skill and Competency in the Operating Room. , 2017, Annual review of biomedical engineering.

[9]  L. Maier-Hein,et al.  Methods and open-source toolkit for analyzing and visualizing challenge results , 2019, Scientific reports.

[10]  J. J. Jakimowicz,et al.  The Eindhoven laparoscopic cholecystectomy training course—improving operating room performance using virtual reality training: results from the first E.A.E.S. accredited virtual reality trainings curriculum , 2005, Surgical Endoscopy And Other Interventional Techniques.

[11]  J. Birkmeyer,et al.  Surgical skill and complication rates after bariatric surgery. , 2013, The New England journal of medicine.

[12]  Lena Maier-Hein,et al.  BIAS: Transparent reporting of biomedical image analysis challenges , 2019, Medical Image Analysis.

[13]  Mika N Sinanan,et al.  Reliable Assessment of Laparoscopic Performance in the Operating Room Using Videotape Analysis , 2007, Surgical innovation.

[14]  Yaozong Gao,et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge , 2019, Medical Image Anal..

[15]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  N. Padoy,et al.  Artificial Intelligence for Surgical Safety , 2020, Annals of surgery.

[17]  N. Padoy,et al.  A Computer Vision Platform to Automatically Locate Critical Events in Surgical Videos: Documenting Safety in Laparoscopic Cholecystectomy. , 2021, Annals of surgery.

[18]  Thomas M. Ward,et al.  SAGES consensus recommendations on an annotation framework for surgical video , 2021, Surgical Endoscopy.

[19]  Andreas Bihlmaier,et al.  Learning dynamic spatial relations: the case of a knowledge-based endoscopic camera guidance robot , 2016 .

[20]  R. Tubbs,et al.  The clinical anatomy of cystic artery variations: a review of over 9800 cases , 2016, Surgical and Radiologic Anatomy.

[21]  Su-Lin Lee,et al.  Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis , 2017, Lecture Notes in Computer Science.

[22]  Andru Putra Twinanda,et al.  EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos , 2016, IEEE Transactions on Medical Imaging.

[23]  T. Grantcharov,et al.  Randomized clinical trial of virtual reality simulation for laparoscopic skills training , 2004, The British journal of surgery.

[24]  J. L. Willems,et al.  The diagnostic performance of computer programs for the interpretation of electrocardiograms. , 1992, The New England journal of medicine.

[25]  S. Speidel,et al.  Machine Learning for Surgical Phase Recognition: A Systematic Review. , 2020, Annals of surgery.

[26]  C. Greenberg,et al.  Association Between Surgeon Technical Skills and Patient Outcomes. , 2020, JAMA surgery.

[27]  Constantinos Loukas,et al.  Video content analysis of surgical procedures , 2018, Surgical Endoscopy.

[28]  Gregory D. Hager,et al.  Impact of data on generalization of AI for surgical intelligence applications , 2020, Scientific Reports.

[29]  D. Hashimoto,et al.  Safe cholecystectomy multi-society practice guideline and state-of-the-art consensus conference on prevention of bile duct injury during cholecystectomy , 2020, Surgical Endoscopy.

[30]  Pietro Piazzolla,et al.  Intraoperative surgery room management: A deep learning perspective , 2020, The international journal of medical robotics + computer assisted surgery : MRCAS.

[31]  Martin Wagner,et al.  Active learning using deep Bayesian networks for surgical workflow analysis , 2018, International Journal of Computer Assisted Radiology and Surgery.

[32]  Andru Putra Twinanda,et al.  Deep Neural Networks Predict Remaining Surgery Duration from Cholecystectomy Videos , 2017, MICCAI.

[33]  Anvil , 2021, Encyclopedic Dictionary of Archaeology.

[34]  Nils Strodthoff,et al.  Detecting and interpreting myocardial infarction using fully convolutional neural networks , 2018, Physiological measurement.

[35]  Stefanie Speidel,et al.  Video-based surgical skill assessment using 3D convolutional neural networks , 2019, International Journal of Computer Assisted Radiology and Surgery.

[36]  B. Peng,et al.  Comments on "Situating Artificial Intelligence in Surgery, a Focus on Disease Severity". , 2021, Annals of surgery.

[37]  Jeffrey A. Golden,et al.  Deep Learning Algorithms for Detection of Lymph Node Metastases From Breast Cancer: Helping Artificial Intelligence Be Seen. , 2017, JAMA.

[38]  S. Speidel,et al.  A learning robot for cognitive camera control in minimally invasive surgery , 2021, Surgical Endoscopy.

[39]  Satoshi Kondo,et al.  CATARACTS: Challenge on automatic tool annotation for cataRACT surgery , 2019, Medical Image Anal..

[40]  Thomas M. Ward,et al.  Automated operative phase identification in peroral endoscopic myotomy , 2020, Surgical Endoscopy.

[41]  Irfan A. Essa,et al.  Automated surgical skill assessment in RMIS training , 2017, International Journal of Computer Assisted Radiology and Surgery.

[42]  Wei Wei,et al.  Thoracic Disease Identification and Localization with Limited Supervision , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Chi-Wing Fu,et al.  SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network , 2018, IEEE Transactions on Medical Imaging.

[44]  Aaron Carass,et al.  Why rankings of biomedical image analysis competitions should be interpreted with care , 2018, Nature Communications.

[45]  Ziheng Wang,et al.  Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery , 2018, International Journal of Computer Assisted Radiology and Surgery.

[46]  J. Doyle,et al.  A universal global rating scale for the evaluation of technical skills in the operating room. , 2007, American journal of surgery.

[47]  Robert G. Radwin,et al.  Modeling Surgical Technical Skill Using Expert Assessment for Automated Computer Rating , 2017, Annals of surgery.

[48]  Gregory D. Hager,et al.  A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery , 2017, IEEE Transactions on Biomedical Engineering.

[49]  Germain Forestier,et al.  Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks , 2019, International Journal of Computer Assisted Radiology and Surgery.

[50]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[51]  G. Rosman,et al.  Computer Vision Analysis of Intraoperative Video: Automated Recognition of Operative Steps in Laparoscopic Sleeve Gastrectomy. , 2019, Annals of surgery.

[52]  Nicolas Martin,et al.  Assisted phase and step annotation for surgical videos , 2020, International Journal of Computer Assisted Radiology and Surgery.

[53]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[54]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[56]  Melina C Vassiliou,et al.  A global assessment tool for evaluation of intraoperative laparoscopic skills. , 2005, American journal of surgery.

[57]  Ryan S. Decker,et al.  Supervised autonomous robotic soft tissue surgery , 2016, Science Translational Medicine.

[58]  Mathias Unberath,et al.  CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions , 2019, Proceedings of the IEEE.

[59]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[60]  Sebastian Bodenstedt,et al.  Context-aware Augmented Reality in laparoscopic surgery , 2013, Comput. Medical Imaging Graph..

[61]  Lena Maier-Hein,et al.  How can we learn (more) from challenges? A statistical approach to driving future algorithm development , 2021, ArXiv.

[62]  Тараса Шевченка,et al.  Quo vadis? , 2013, Clinical chemistry.

[63]  Yuanyuan Wang,et al.  A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging , 2020, Medical Image Anal..

[64]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Martin Wagner,et al.  Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data , 2018, International Journal of Computer Assisted Radiology and Surgery.

[66]  Eui Jin Hwang,et al.  Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. , 2019, Radiology.

[67]  Ramandeep Singh,et al.  Deep learning in chest radiography: Detection of findings and presence of change , 2018, PloS one.

[68]  Gero Strauß,et al.  Research Paper: Validation of Knowledge Acquisition for Surgical Process Models , 2009, J. Am. Medical Informatics Assoc..

[69]  Luc Van Gool,et al.  Temporal Segment Networks: Towards Good Practices for Deep Action Recognition , 2016, ECCV.

[70]  Douglas A Wiegmann,et al.  A Statewide Surgical Coaching Program Provides Opportunity for Continuous Professional Development , 2017, Annals of surgery.

[71]  Eric J Topol,et al.  High-performance medicine: the convergence of human and artificial intelligence , 2019, Nature Medicine.

[72]  J. Donohue.  Laparoscopic cholecystectomy. , 1992, Surgery.

[73]  Jeanette W. Chung,et al.  Association Between Surgical Technical Skill and Long-term Survival for Colon Cancer. , 2020, JAMA oncology.

[74]  J. Kiefer,et al.  Stochastic Estimation of the Maximum of a Regression Function , 1952 .

[75]  Zhuowen Tu,et al.  Aggregated Residual Transformations for Deep Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).