Computer Vision-Assisted Surgery: Real-Time Instrument Tracking with Machine Learning

Visual tracking of surgical instruments is a key component of various computer-assisted interventions, yet a very challenging problem in the field of Computer Vision. This dissertation presents novel approaches which leverage machine learning techniques for precise real-time tracking and 2D pose estimation of instruments. The achieved results demonstrate that the proposed methods based on Random Forests and Deep Learning provide remarkable advantages with respect to the state of the art in terms of accuracy, robustness and generalization.

[1]  G. Peyman,et al.  Vitreoretinal Surgical Techniques , 2001 .

[2]  Bernd J Pichler,et al.  Multimodal imaging approaches: PET/CT and PET/MRI. , 2008, Handbook of experimental pharmacology.

[3]  Andrew M. Rollins,et al.  Integrative Advances for OCT-Guided Ophthalmic Surgery and Intraoperative OCT: Microscope Integration, Surgical Instrumentation, and Heads-Up Display Surgeon Feedback , 2014, 2015 Conference on Lasers and Electro-Optics (CLEO).

[4]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[5]  Y. Nose,et al.  Prevalence and risk factors for epiretinal membranes in a Japanese population: the Hisayama study , 2003, Graefe's Archive for Clinical and Experimental Ophthalmology.

[6]  Russell H. Taylor,et al.  Data-Driven Visual Tracking in Retinal Microsurgery , 2012, MICCAI.

[7]  J. Ponsky,et al.  Endoluminal surgery: past, present and future , 2006, Surgical Endoscopy And Other Interventional Techniques.

[8]  Pierre Jannin,et al.  Automatic data-driven real-time segmentation and recognition of surgical workflow , 2016, International Journal of Computer Assisted Radiology and Surgery.

[9]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[10]  Guang-Zhong Yang,et al.  Removing specular reflection components for robotic assisted laparoscopic surgery , 2005, IEEE International Conference on Image Processing 2005.

[11]  Austin Reiter,et al.  Feature Classification for Tracking Articulated Surgical Tools , 2012, MICCAI.

[12]  J. Bennett,et al.  Gene Therapy and Stem Cell Transplantation in Retinal Disease: The New Frontier. , 2016, Ophthalmology.

[13]  Kouichi Sakurai,et al.  One Pixel Attack for Fooling Deep Neural Networks , 2017, IEEE Transactions on Evolutionary Computation.

[14]  Nassir Navab,et al.  CFCM: Segmentation via Coarse to Fine Context Memory , 2018, MICCAI.

[15]  Russell H. Taylor,et al.  Intraoperative Visualization of Anatomical Targets in Retinal Surgery , 2008, 2008 IEEE Workshop on Applications of Computer Vision.

[16]  J. F. Vander.  Value of internal limiting membrane peeling in surgery for idiopathic macular hole and the correlation between function and retinal morphology , 2010 .

[17]  C. Simopoulos,et al.  Twenty Years of Laparoscopic Cholecystectomy: Philippe Mouret—March 17, 1987 , 2008, JSLS : Journal of the Society of Laparoendoscopic Surgeons.

[18]  Derek Hoiem,et al.  Learning without Forgetting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  George L. Spaeth,et al.  Ophthalmic surgery, principles and practice , 1982 .

[20]  J. Sipperley,et al.  Macular pucker removal with and without internal limiting membrane peeling: pilot study. , 2003, Ophthalmology.

[21]  K. Cleary,et al.  Image-guided interventions: technology review and clinical applications. , 2010, Annual review of biomedical engineering.

[22]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[23]  Peter K Kaiser,et al.  Intraoperative optical coherence tomography using the RESCAN 700: preliminary results from the DISCOVER study , 2014, British Journal of Ophthalmology.

[24]  J.B. Christoforidis,et al.  Retinal Damage Induced by Internal Limiting Membrane Removal , 2015, Journal of ophthalmology.

[25]  David Joseph Tan,et al.  Multi-forest Tracker: A Chameleon in Tracking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Narendran,et al.  Principles and Practice of Vitreoretinal Surgery , 2014 .

[27]  Sébastien Ourselin,et al.  Real-Time Segmentation of Non-rigid Surgical Tools Based on Deep Learning and Tracking , 2016, CARE@MICCAI.

[28]  Pierre Jannin,et al.  Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures , 2012, International Journal of Computer Assisted Radiology and Surgery.

[29]  Heinz Wörn,et al.  Workflow analysis and surgical phase recognition in minimally invasive surgery , 2012, 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[30]  Stefanie Speidel,et al.  Tracking of Instruments in Minimally Invasive Surgery for Surgical Skill Analysis , 2006, MIAR.

[31]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[32]  Sébastien Ourselin,et al.  Image Based Surgical Instrument Pose Estimation with Multi-class Labelling and Optical Flow , 2015, MICCAI.

[33]  Gyslain Giguère,et al.  Limits in decision making arise from limits in memory retrieval , 2013, Proceedings of the National Academy of Sciences.

[34]  Bernt Schiele,et al.  Detecting Surgical Tools by Modelling Local Appearance and Global Shape , 2015, IEEE Transactions on Medical Imaging.

[35]  Russell H. Taylor,et al.  Unified Detection and Tracking in Retinal Microsurgery , 2011, MICCAI.

[36]  Tinne Tuytelaars,et al.  Expert Gate: Lifelong Learning with a Network of Experts , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  G. Barbash,et al.  New technology and health care costs--the case of robot-assisted surgery. , 2010, The New England journal of medicine.

[38]  Nassir Navab,et al.  Statistical modeling and recognition of surgical workflow , 2012, Medical Image Anal..

[39]  J Dankelman,et al.  Systems for tracking minimally invasive surgical instruments , 2007, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[40]  M. Mack,et al.  Minimally invasive and robotic surgery. , 2001, JAMA.

[41]  Pascal Fua,et al.  Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery , 2017, MICCAI.

[42]  Shahram Payandeh,et al.  Visual Tracking of Laparoscopic Instruments , 2014 .

[43]  Nassir Navab,et al.  CRF-Based Model for Instrument Detection and Pose Estimation in Retinal Microsurgery , 2016, Comput. Math. Methods Medicine.

[44]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Nassir Navab,et al.  Concurrent Segmentation and Localization for Tracking of Surgical Instruments , 2017, MICCAI.

[46]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[47]  Nassir Navab,et al.  Ultrasound interactive segmentation with tensor-graph methods , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[48]  Nassir Navab,et al.  Deeper Depth Prediction with Fully Convolutional Residual Networks , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[49]  Henry C. Lin,et al.  Review of methods for objective surgical skill evaluation , 2011, Surgical Endoscopy.

[50]  Christoph Tappeiner,et al.  Retinal slit lamp video mosaicking , 2016, International Journal of Computer Assisted Radiology and Surgery.

[51]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[52]  S. Haykin.  Kalman Filtering and Neural Networks , 2001 .

[53]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[54]  Russell H. Taylor,et al.  Visual Tracking of Surgical Tools for Proximity Detection in Retinal Surgery , 2011, IPCAI.

[55]  Nassir Navab,et al.  Automatic Initialization and Failure Detection for Surgical Tool Tracking in Retinal Microsurgery , 2017, Bildverarbeitung für die Medizin.

[56]  Kevin Cleary,et al.  OR2020 workshop overview: operating room of the future , 2004, CARS.

[57]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[58]  A. Harrell,et al.  Minimally invasive abdominal surgery: lux et veritas past, present, and future. , 2005, American journal of surgery.

[59]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[60]  Cordelia Schmid,et al.  Incremental Learning of Object Detectors without Catastrophic Forgetting , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[61]  Mamoru Mitsuishi,et al.  Full state visual forceps tracking under a microscope using projective contour models , 2012, 2012 IEEE International Conference on Robotics and Automation.

[62]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[63]  Wolfgang Drexler,et al.  Introduction to OCT , 2015 .

[64]  Jason J. Corso,et al.  Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection , 2017, IEEE Transactions on Medical Imaging.

[65]  A. Lanfranco,et al.  Robotic Surgery: A Current Perspective , 2004, Annals of surgery.

[66]  P. Allen,et al.  Marker-less Articulated Surgical Tool Detection , 2012 .

[67]  Gregory D. Hager,et al.  Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation , 2016, ECCV.

[68]  R. Weale,et al.  The Epidemiology of Eye Disease , 1998 .

[69]  Nassir Navab,et al.  Real-Time Online Adaption for Robust Instrument Tracking and Pose Estimation , 2016, MICCAI.

[70]  Nassir Navab,et al.  Introducing Augmented Reality to Optical Coherence Tomography in Ophthalmic Microsurgery , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[71]  Tien Yin Wong,et al.  Prevalence and risk factors for epiretinal membranes in a multi-ethnic United States population. , 2011, Ophthalmology.

[72]  Nassir Navab,et al.  Deep Residual Learning for Instrument Segmentation in Robotic Surgery , 2017, MLMI@MICCAI.

[73]  Nassir Navab,et al.  Random Forests for Phase Detection in Surgical Workflow Analysis , 2014, IPCAI.

[74]  S. Srivastava,et al.  Intraoperative optical coherence tomography-compatible surgical instruments for real-time image-guided ophthalmic surgery , 2017, British Journal of Ophthalmology.

[75]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[76]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[77]  Wolfgang Drexler,et al.  State-of-the-art retinal optical coherence tomography , 2008, Progress in Retinal and Eye Research.

[78]  Jamie L. Reese,et al.  Outcomes of Intraoperative OCT-Assisted Epiretinal Membrane Surgery from the PIONEER Study. , 2017, Ophthalmology. Retina.

[79]  Kutluay Uluç,et al.  Operating microscopes: past, present, and future. , 2009, Neurosurgical focus.

[80]  Luc Soler,et al.  The status of augmented reality in laparoscopic surgery as of 2016 , 2017, Medical Image Anal..

[81]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[82]  Russell H. Taylor,et al.  Adaptive Multispectral Illumination for Retinal Microsurgery , 2010, MICCAI.

[83]  Pascal Fua,et al.  Fast Part-Based Classification for Instrument Detection in Minimally Invasive Surgery , 2014, MICCAI.

[84]  Mark R. Pickering,et al.  A new multi-modal similarity measure for fast gradient-based 2D-3D image registration , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[85]  Austin Reiter,et al.  An online learning approach to in-vivo tracking using synergistic features , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[86]  David A. Wagner,et al.  Towards Evaluating the Robustness of Neural Networks , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[87]  J. Barraquer,et al.  The history of the microscope in ocular surgery , 1980, Journal of microsurgery.

[88]  Jie Liu,et al.  A Robust Mosaicing Method with Super-Resolution for Optical Medical Images , 2010, MIAR.

[89]  Raphael Sznitman,et al.  Pathological OCT Retinal Layer Segmentation Using Branch Residual U-Shape Networks , 2017, MICCAI.

[90]  Nassir Navab,et al.  ReLayNet: Retinal Layer and Fluid Segmentation of Macular Optical Coherence Tomography using Fully Convolutional Network , 2017, Biomedical optics express.

[91]  Justis P. Ehlers,et al.  Intraoperative optical coherence tomography: past, present, and future , 2016, Eye.

[92]  Junzhou Huang,et al.  Instrument Tracking via Online Learning in Retinal Microsurgery , 2014, MICCAI.

[93]  Christoph H. Lampert,et al.  iCaRL: Incremental Classifier and Representation Learning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[94]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[95]  J. Romanelli,et al.  Single-port laparoscopic surgery: an overview , 2009, Surgical Endoscopy.

[96]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[97]  Yi Yang,et al.  Active Multispectral Illumination and Image Fusion for Retinal Microsurgery , 2010, IPCAI.

[98]  Nassir Navab,et al.  Holistic Human Pose Estimation with Regression Forests , 2014, AMDO.

[99]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.

[100]  Constantinos Loukas,et al.  Video content analysis of surgical procedures , 2018, Surgical Endoscopy.

[101]  Danail Stoyanov,et al.  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, IEEE Transactions on Medical Imaging.

[102]  T. Baron,et al.  Natural orifice transluminal endoscopic surgery , 2007, The British journal of surgery.

[103]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[104]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[105]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.