A Real-time Augmented Reality Surgical System for Overlaying Stiffness Information

We describe a surgical system that autonomously searches for tumors and dynamically displays a computer graphic model of them super-imposed on the organ (or in our case, phantom). Once localized, the phantom is tracked in real time and augmented with overlaid stiffness information in 3D. We believe that such a system has the potential to quickly reveal the location and shape of tumors, and the visual overlay will reduce the cognitive overload of the surgeon. The contribution of this paper is the integration of disparate technologies to achieve this system. In fact, to the best of our knowledge, our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on a phantom prostate model and other silicone organs with embedded stiff inclusions using the da Vinci research kit (dVRK).

[1]  Alkis Gotovos,et al.  Active Learning for Level Set Estimation , 2022 .

[2]  Paul J. Besl,et al.  Method for registration of 3-D shapes , 1992, Other Conferences.

[3]  Howie Choset,et al.  Dynamic Texture Mapping of 3 D models for Stiffness Map Visualization , 2022 .

[4]  Peter Kazanzides,et al.  Force-Controlled Exploration for Updating Virtual Fixture Geometry in Model-Mediated Telemanipulation , 2017 .

[5]  Kenneth Y. Goldberg,et al.  Tumor localization using automated palpation with Gaussian Process Adaptive Sampling , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[6]  Nabil Simaan,et al.  Algorithms for autonomous exploration and estimation in compliant environments , 2012, Robotica.

[7]  Peter Kazanzides,et al.  A Computational Framework for Complementary Situational Awareness (CSA) in Surgical Assistant Robots , 2018, 2018 Second IEEE International Conference on Robotic Computing (IRC).

[8]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Howie Choset,et al.  Bingham Distribution-Based Linear Filter for Online Pose Estimation , 2017, Robotics: Science and Systems.

[10]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[11]  Allison M. Okamura,et al.  Methods to Segment Hard Inclusions in Soft Tissue During Autonomous Robotic Palpation , 2015, IEEE Transactions on Robotics.

[12]  Rajnikant V. Patel,et al.  Robot-assisted Tactile Sensing for Minimally Invasive Tumor Localization , 2009, Int. J. Robotics Res..

[13]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[14]  Purang Abolmaesumi,et al.  Point-Based Rigid-Body Registration Using an Unscented Kalman Filter , 2007, IEEE Transactions on Medical Imaging.

[15]  Peter Kazanzides,et al.  Updating Virtual Fixtures From Exploration Data in Force-controlled Model-based Telemanipulation , 2016 .

[16]  Chen Yang,et al.  Development of an inexpensive tri-axial force sensor for minimally invasive surgery , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Howie Choset,et al.  Estimating SE(3) elements using a dual quaternion based linear Kalman filter , 2016, Robotics: Science and Systems.

[18]  Nigel H. Lovell,et al.  A review of tactile sensing technologies with applications in biomedical engineering , 2012 .

[19]  Michael D. Naish,et al.  A Breakthrough in Tumor Localization: Combining Tactile Sensing and Ultrasound to Improve Tumor Localization in Robotics-Assisted Minimally Invasive Surgery , 2017, IEEE Robotics & Automation Magazine.

[20]  J. Shuhaiber,et al.  Augmented reality in surgery. , 2004, Archives of surgery.

[21]  J. Marescaux,et al.  Augmented-reality-assisted laparoscopic adrenalectomy. , 2004, JAMA.

[22]  Howie Choset,et al.  Probabilistic pose estimation using a Bingham distribution-based linear filter , 2018, Int. J. Robotics Res..

[23]  V. Ozben,et al.  Robotic-Assisted Minimally Invasive Surgery , 2019, Springer International Publishing.

[24]  Allison M. Okamura,et al.  Tissue property estimation and graphical display for teleoperated robot-assisted surgery , 2009, 2009 IEEE International Conference on Robotics and Automation.

[25]  L.D. Seneviratne,et al.  State-of-the-Art in Force and Tactile Sensing for Minimally Invasive Surgery , 2008, IEEE Sensors Journal.

[26]  Howie Choset,et al.  Utility-Guided Palpation for Locating Tissue Abnormalities , 2017, IEEE Robotics and Automation Letters.

[27]  Robert D. Howe,et al.  Remote palpation technology , 1995 .

[28]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[29]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Kaspar Althoefer,et al.  Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery , 2010, IEEE Transactions on Biomedical Engineering.

[31]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[32]  Andriy Myronenko,et al.  Point Set Registration: Coherent Point Drift , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Nazim Haouchine,et al.  Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[34]  Russell H. Taylor,et al.  Preliminary Evaluation of an Online Estimation Method for Organ Geometry and Tissue Stiffness , 2018, IEEE Robotics and Automation Letters.

[35]  Lu Li,et al.  A surgical system for automatic registration, stiffness mapping and dynamic image overlay , 2018, 2018 International Symposium on Medical Robotics (ISMR).

[36]  Russell H. Taylor,et al.  Concurrent nonparametric estimation of organ geometry and tissue stiffness using continuous adaptive palpation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[37]  Howie Choset,et al.  Using Bayesian optimization to guide probing of a flexible environment for simultaneous registration and stiffness mapping , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[38]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[39]  Howie Choset,et al.  Simultaneous Compliance and Registration Estimation for Robotic Surgery , 2014, Robotics: Science and Systems.

[40]  Allison M. Okamura,et al.  A single-use haptic palpation probe for locating subcutaneous blood vessels in robot-assisted minimally invasive surgery , 2015, 2015 IEEE International Conference on Automation Science and Engineering (CASE).

[41]  Howie Choset,et al.  Trajectory-Optimized Sensing for Active Search of Tissue Abnormalities in Robotic Surgery , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[42]  Howie Choset,et al.  Complementary model update: A method for simultaneous registration and stiffness mapping in flexible environments , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[43]  Russell H. Taylor,et al.  Iterative Most-Likely Point Registration (IMLP): A Robust Algorithm for Computing Optimal Shape Alignment , 2015, PloS one.