Optical Coherence Tomography-Guided Robotic Ophthalmic Microsurgery via Reinforcement Learning from Demonstration

Ophthalmic microsurgery is technically difficult because the scale of required surgical tool manipulations challenge the limits of the surgeon's visual acuity, sensory perception, and physical dexterity. Intraoperative optical coherence tomography (OCT) imaging with micrometer-scale resolution is increasingly being used to monitor and provide enhanced real-time visualization of ophthalmic surgical maneuvers, but surgeons still face physical limitations when manipulating instruments inside the eye. Autonomously controlled robots are one avenue for overcoming these physical limitations. In this article, we demonstrate the feasibility of using learning from demonstration and reinforcement learning with an industrial robot to perform OCT-guided corneal needle insertions in an ex vivo model of deep anterior lamellar keratoplasty (DALK) surgery. Our reinforcement learning agent trained on ex vivo human corneas, then outperformed surgical fellows in reaching a target needle insertion depth in mock corneal surgery trials. This article shows the combination of learning from demonstration and reinforcement learning is a viable option for performing OCT-guided robotic ophthalmic surgery.

[1]  Mark J Mannis,et al.  Corneal Graft Rejection 10 Years After Penetrating Keratoplasty in the Cornea Donor Study , 2014, Cornea.

[2]  Demis Hassabis,et al.  Mastering the game of Go without human knowledge , 2017, Nature.

[3]  Ronald R Krueger,et al.  Outcomes of Deep Anterior Lamellar Keratoplasty for Keratoconus: Learning Curve and Advantages of the Big Bubble Technique , 2012, Cornea.

[4]  Iulian Iordachita,et al.  A Submillimetric 3-DOF Force Sensing Instrument With Integrated Fiber Bragg Grating for Retinal Microsurgery , 2014, IEEE Transactions on Biomedical Engineering.

[5]  Nabil Simaan,et al.  Evaluation of microsurgical tasks with OCT-guided and/or robot-assisted ophthalmic forceps. , 2015, Biomedical optics express.

[6]  Ashutosh Tewari,et al.  Laparoscopic and robot assisted radical prostatectomy: establishment of a structured program and preliminary analysis of outcomes. , 2002, The Journal of urology.

[7]  Byron Boots,et al.  Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning , 2018, ICLR.

[8]  Massimo Busin,et al.  Anterior segment optical coherence tomography-guided big-bubble technique. , 2013, Ophthalmology.

[9]  Daniel Glozman,et al.  Image-Guided Robotic Flexible Needle Steering , 2007, IEEE Transactions on Robotics.

[10]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[11]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[12]  Sergey Levine,et al.  End-to-End Training of Deep Visuomotor Policies , 2015, J. Mach. Learn. Res..

[13]  Marco Caversaccio,et al.  Instrument flight to the inner ear , 2017, Science Robotics.

[14]  Colm McAlinden,et al.  Subjective quality of vision before and after cataract surgery. , 2012, Archives of ophthalmology.

[15]  Alec Radford,et al.  Proximal Policy Optimization Algorithms , 2017, ArXiv.

[16]  Yuankai K. Tao,et al.  Microscope-integrated intraoperative OCT with electrically tunable focus and heads-up display for imaging of ophthalmic surgical maneuvers. , 2014, Biomedical optics express.

[17]  R. Bellman A Markovian Decision Process , 1957 .

[18]  Tom Schaul,et al.  Deep Q-learning From Demonstrations , 2017, AAAI.

[19]  Cameron N. Riviere,et al.  Three-dimensional accuracy assessment of eye surgeons , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[20]  Zerui Wang,et al.  Toward Semi-autonomous Cryoablation of Kidney Tumors via Model-Independent Deformable Tissue Manipulation Technique , 2018, Annals of Biomedical Engineering.

[21]  Mohammed Anwar,et al.  Big‐bubble technique to bare Descemet's membrane in anterior lamellar keratoplasty , 2002, Journal of cataract and refractive surgery.

[22]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[23]  Guy Lever,et al.  Deterministic Policy Gradient Algorithms , 2014, ICML.

[24]  R Bellman,et al.  On the Theory of Dynamic Programming. , 1952, Proceedings of the National Academy of Sciences of the United States of America.

[25]  Joseph A. Izatt,et al.  Real-time corneal segmentation and 3D needle tracking in intrasurgical OCT , 2018, Biomedical optics express.

[26]  E A Archila,et al.  Deep lamellar keratoplasty dissection of host tissue with intrastromal air injection. , 1984, Cornea.

[27]  Blake Hannaford,et al.  Semi-autonomous simulated brain tumor ablation with RAVENII Surgical Robot using behavior tree , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Axel Krieger,et al.  Semi-autonomous electrosurgery for tumor resection using a multi-degree of freedom electrosurgical tool and visual servoing , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[29]  Pieter Abbeel,et al.  Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations , 2010, 2010 IEEE International Conference on Robotics and Automation.

[30]  S. Farsiu,et al.  Live volumetric (4D) visualization and guidance of in vivo human ophthalmic surgery with intraoperative optical coherence tomography , 2016, Scientific Reports.

[31]  Gregory D. Hager,et al.  Motion generation of robotic surgical tasks: Learning from expert demonstrations , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[32]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[33]  Hcm Thijs Meenink Vitreo-retinal eye surgery robot : sustainable precision , 2011 .

[34]  Allison M. Okamura,et al.  3-D Ultrasound-Guided Robotic Needle Steering in Biological Tissue , 2014, IEEE Transactions on Biomedical Engineering.

[35]  Cameron N. Riviere,et al.  Micron: An Actively Stabilized Handheld Tool for Microsurgery , 2012, IEEE Transactions on Robotics.

[36]  Pieter Abbeel,et al.  Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[37]  James A. Young,et al.  Early Experience with Telemanipulative Robot-Assisted Laparoscopic Cholecystectomy Using da Vinci , 2002, Surgical laparoscopy, endoscopy & percutaneous techniques.

[38]  Ron Alterovitz,et al.  High-Frequency Replanning Under Uncertainty Using Parallel Sampling-Based Motion Planning , 2015, IEEE Transactions on Robotics.

[39]  G. Ripandelli,et al.  Optical coherence tomography. , 1998, Seminars in ophthalmology.

[40]  David Silver,et al.  Deep Reinforcement Learning with Double Q-Learning , 2015, AAAI.

[41]  Jin Seob Kim,et al.  Nonholonomic Modeling of Needle Steering , 2006, Int. J. Robotics Res..

[42]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[43]  Sergey Levine,et al.  Trust Region Policy Optimization , 2015, ICML.

[44]  Peter Kazanzides,et al.  An image-directed robotic system for precise orthopaedic surgery , 1994, IEEE Trans. Robotics Autom..

[45]  Jin U. Kang,et al.  Active tremor cancellation by a "smart" handheld vitreoretinal microsurgical tool using swept source optical coherence tomography. , 2012, Optics express.

[46]  Patrick S. Jensen,et al.  Surgical Forces and Tactile Perception During Retinal Microsurgery , 1999, MICCAI.

[47]  Marcin Andrychowicz,et al.  Overcoming Exploration in Reinforcement Learning with Demonstrations , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[48]  D F P Larkin,et al.  Corneal Transplantation: The Forgotten Graft , 2004, American journal of transplantation : official journal of the American Society of Transplantation and the American Society of Transplant Surgeons.

[49]  Welch Bl THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED , 1947 .

[50]  A. Farmery,et al.  First-in-human study of the safety and viability of intraocular robotic surgery , 2018, Nature Biomedical Engineering.

[51]  H. Dua,et al.  Outcomes of deep anterior lamellar keratoplasty following successful and failed ‘big bubble’ , 2011, British Journal of Ophthalmology.

[52]  W. R. Buckland,et al.  Contributions to Probability and Statistics , 1960 .

[53]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[54]  Stefan Schaal,et al.  Robot Learning From Demonstration , 1997, ICML.

[55]  Thomas Gaujoux,et al.  Long-term results of deep anterior lamellar versus penetrating keratoplasty. , 2012, Ophthalmology.

[56]  Ryan S. Decker,et al.  Supervised autonomous robotic soft tissue surgery , 2016, Science Translational Medicine.

[57]  C.N. Riviere,et al.  Static and dynamic accuracy of vitreoretinal surgeons , 2004, The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[58]  John A. Nelder,et al.  A Simplex Method for Function Minimization , 1965, Comput. J..

[59]  A. Hofman,et al.  Distribution of central corneal thickness and its association with intraocular pressure: The Rotterdam Study. , 1997, American journal of ophthalmology.

[60]  Anthony N Kuo,et al.  Needle Depth and Big-Bubble Success in Deep Anterior Lamellar Keratoplasty: An Ex Vivo Microscope-Integrated OCT Study , 2016, Cornea.

[61]  Sergey Levine,et al.  Collective robot reinforcement learning with distributed asynchronous guided policy search , 2016, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[62]  Martin Culjat,et al.  FEASIBILITY STUDY OF INTRAOCULAR ROBOTIC SURGERY WITH THE da VINCI SURGICAL SYSTEM , 2008, Retina.

[63]  Susanne Binder,et al.  FEASIBILITY OF INTRASURGICAL SPECTRAL-DOMAIN OPTICAL COHERENCE TOMOGRAPHY , 2011, Retina.

[64]  Martin A. Riedmiller,et al.  Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards , 2017, ArXiv.