Artificial Intelligence-Assisted Surgery: Potential and Challenges

Background: Artificial intelligence (AI) has recently achieved considerable success in different domains including medical applications. Although current advances are expected to impact surgery, up until now AI has not been able to leverage its full potential due to several challenges that are specific to that field. Summary: This review summarizes data-driven methods and technologies needed as a prerequisite for different AI-based assistance functions in the operating room. Potential effects of AI usage in surgery will be highlighted, concluding with ongoing challenges to enabling AI for surgery. Key Messages: AI-assisted surgery will enable data-driven decision-making via decision support systems and cognitive robotic assistance. The use of AI for workflow analysis will help provide appropriate assistance in the right context. The requirements for such assistance must be defined by surgeons in close cooperation with computer scientists and engineers. Once the existing challenges will have been solved, AI assistance has the potential to improve patient care by supporting the surgeon without replacing him or her.

[1]  Suman V. Ravuri,et al.  A Clinically Applicable Approach to Continuous Prediction of Future Acute Kidney Injury , 2019, Nature.

[2]  Martin Wagner,et al.  Active learning using deep Bayesian networks for surgical workflow analysis , 2018, International Journal of Computer Assisted Radiology and Surgery.

[3]  Nassir Navab,et al.  On-line Recognition of Surgical Activity for Monitoring in the Operating Room , 2008, AAAI.

[4]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[5]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[6]  M. P. Mulder,et al.  Effect of a Machine Learning-Derived Early Warning System for Intraoperative Hypotension vs Standard Care on Depth and Duration of Intraoperative Hypotension During Elective Noncardiac Surgery: The HYPE Randomized Clinical Trial. , 2020, JAMA.

[7]  Stefanie Speidel,et al.  Non-Rigid Volume to Surface Registration using a Data-Driven Biomechanical Model , 2020, MICCAI.

[8]  Danail Stoyanov,et al.  CaDIS: Cataract Dataset for Image Segmentation , 2019, ArXiv.

[9]  Gregory D. Hager,et al.  An Improved Model for Segmentation and Recognition of Fine-Grained Activities with Application to Surgical Training Tasks , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[10]  Andreas Holzinger,et al.  From Machine Learning to Explainable AI , 2018, 2018 World Symposium on Digital Intelligence for Systems and Machines (DISA).

[11]  Andru Putra Twinanda,et al.  RSDNet: Learning to Predict Remaining Surgery Duration from Laparoscopic Videos Without Manual Annotations , 2018, IEEE Transactions on Medical Imaging.

[12]  Andre Esteva,et al.  A guide to deep learning in healthcare , 2019, Nature Medicine.

[13]  Sebastian Bodenstedt,et al.  Unsupervised Temporal Video Segmentation as an Auxiliary Task for Predicting the Remaining Surgery Duration , 2019, OR/MLCN@MICCAI.

[14]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.

[15]  Thomas Neumuth,et al.  Ontology-based specification, identification and analysis of perioperative risks , 2017, Journal of Biomedical Semantics.

[16]  Kevin Donnelly,et al.  SNOMED-CT: The advanced terminology and coding system for eHealth. , 2006, Studies in health technology and informatics.

[17]  C. Martin.  2015 , 2015, Les 25 ans de l’OMC: Une rétrospective en photos.

[18]  Thomas Neumuth,et al.  Ontological Modelling of Surgical Knowledge , 2009, GI Jahrestagung.

[19]  Jason J. Corso,et al.  Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection , 2017, IEEE Transactions on Medical Imaging.

[20]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[21]  Rüdiger Dillmann,et al.  LapOntoSPM: an ontology for laparoscopic surgeries and its application to surgical phase recognition , 2015, International Journal of Computer Assisted Radiology and Surgery.

[22]  Russell H. Taylor,et al.  Surgical data science for next-generation interventions , 2017, Nature Biomedical Engineering.

[23]  Martin Wagner,et al.  Prediction of laparoscopic procedure duration using unlabeled, multimodal sensor data , 2018, International Journal of Computer Assisted Radiology and Surgery.

[24]  Klaus H. Maier-Hein,et al.  Exploiting the potential of unlabeled endoscopic video data with self-supervised learning , 2017, International Journal of Computer Assisted Radiology and Surgery.

[25]  Andru Putra Twinanda,et al.  Vision-based approaches for surgical activity recognition using laparoscopic and RBGD videos. (Approches basées vision pour la reconnaissance d'activités chirurgicales à partir de vidéos laparoscopiques et multi-vues RGBD) , 2017 .

[26]  Jin-Young Jang,et al.  International validation and update of the Amsterdam model for prediction of survival after pancreatoduodenectomy for pancreatic cancer. , 2019, European journal of surgical oncology : the journal of the European Society of Surgical Oncology and the British Association of Surgical Oncology.

[27]  Jeremy M. Grimshaw,et al.  Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes , 2019, JAMA network open.

[28]  Michael V. McConnell,et al.  Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning , 2017, Nature Biomedical Engineering.

[29]  Andru Putra Twinanda,et al.  EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos , 2016, IEEE Transactions on Medical Imaging.

[30]  Sebastian Bodenstedt,et al.  Image-based laparoscopic bowel measurement , 2016, International Journal of Computer Assisted Radiology and Surgery.

[31]  Lee W. White,et al.  Crowdsourcing to Assess Surgical Skill. , 2015, JAMA surgery.

[32]  Mathias Unberath,et al.  CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions , 2019, Proceedings of the IEEE.

[33]  Henry C. Lin,et al.  JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ) : A Surgical Activity Dataset for Human Motion Modeling , 2014 .

[34]  Ryan S. Decker,et al.  Supervised autonomous robotic soft tissue surgery , 2016, Science Translational Medicine.

[35]  Karsten M. Borgwardt,et al.  Early prediction of circulatory failure in the intensive care unit using machine learning , 2020, Nature Medicine.

[36]  R. DeMatteo,et al.  Can We Predict Incisional Hernia?: Development of a Surgery-specific Decision-Support Interface. , 2019, Annals of surgery.

[37]  David S. Melnick,et al.  International evaluation of an AI system for breast cancer screening , 2020, Nature.

[38]  Stephane Cotin,et al.  Physics-Based Deep Neural Network for Augmented Reality During Liver Surgery , 2019, MICCAI.

[39]  Lena Maier-Hein,et al.  Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation , 2019, MICCAI.

[40]  Riccardo Muradore,et al.  Development of a Cognitive Robotic System for Simple Surgical Tasks , 2015 .

[41]  Klaus H. Maier-Hein,et al.  Crowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence , 2016, MICCAI.

[42]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[43]  Lena Maier-Hein,et al.  Heidelberg Colorectal Data Set for Surgical Data Science in the Sensor Operating Room , 2020, ArXiv.

[44]  Y. Iwashita,et al.  Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy. , 2020, Surgical endoscopy.

[45]  Pierre Jannin,et al.  Surgical process modelling: a review , 2014, International Journal of Computer Assisted Radiology and Surgery.

[46]  Keno März,et al.  Toward a standard ontology of surgical process models , 2018, International Journal of Computer Assisted Radiology and Surgery.

[47]  Stefanie Speidel,et al.  Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks , 2019, International Journal of Computer Assisted Radiology and Surgery.

[48]  Mohammed Ali Hussain,et al.  Patient Health Monitoring Using IoT , 2019 .

[49]  Nassir Navab,et al.  Statistical modeling and recognition of surgical workflow , 2012, Medical Image Anal..

[50]  Nicolas Martin,et al.  Assisted phase and step annotation for surgical videos , 2020, International Journal of Computer Assisted Radiology and Surgery.

[51]  Sebastian Bodenstedt,et al.  Temporal coherence-based self-supervised learning for laparoscopic workflow analysis , 2018, OR 2.0/CARE/CLIP/ISIC@MICCAI.

[52]  Sebastian Bodenstedt,et al.  Rethinking Anticipation Tasks: Uncertainty-aware Anticipation of Sparse Surgical Instrument Usage for Context-aware Assistance , 2020, MICCAI.