A surgical system for automatic registration, stiffness mapping and dynamic image overlay

In this paper we develop a surgical system using the da Vinci research kit (dVRK) that is capable of autonomously searching for tumors and dynamically displaying the tumor location using augmented reality. Such a system has the potential to quickly reveal the location and shape of tumors and visually overlay that information to reduce the cognitive overload of the surgeon. To the best of our knowledge, our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and the shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on phantom prostate models with embedded stiff inclusions.

[1]  Howie Choset,et al.  Development of an Inexpensive Triaxial Force Sensor for Minimally Invasive Surgery , 2017 .

[2]  Allison M. Okamura,et al.  A single-use haptic palpation probe for locating subcutaneous blood vessels in robot-assisted minimally invasive surgery , 2015, 2015 IEEE International Conference on Automation Science and Engineering (CASE).

[3]  Kenneth Y. Goldberg,et al.  Tumor localization using automated palpation with Gaussian Process Adaptive Sampling , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[4]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Howie Choset,et al.  Utility-Guided Palpation for Locating Tissue Abnormalities , 2017, IEEE Robotics and Automation Letters.

[6]  Howie Choset,et al.  Using Bayesian optimization to guide probing of a flexible environment for simultaneous registration and stiffness mapping , 2015, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[7]  Robert D. Howe,et al.  Remote palpation technology , 1995 .

[8]  Nazim Haouchine,et al.  Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Rajnikant V. Patel,et al.  Robot-assisted Tactile Sensing for Minimally Invasive Tumor Localization , 2009, Int. J. Robotics Res..

[10]  Silke Geisen.  Augmented Reality in surgery , 2005 .

[11]  Purang Abolmaesumi,et al.  Point-Based Rigid-Body Registration Using an Unscented Kalman Filter , 2007, IEEE Transactions on Medical Imaging.

[12]  Paul J. Besl,et al.  Method for registration of 3-D shapes , 1992, Other Conferences.

[13]  Howie Choset,et al.  Dynamic Texture Mapping of 3 D models for Stiffness Map Visualization , 2022 .

[14]  Nabil Simaan,et al.  Algorithms for autonomous exploration and estimation in compliant environments , 2012, Robotica.

[15]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[16]  Alkis Gotovos,et al.  Active Learning for Level Set Estimation , 2022 .

[17]  Allison M. Okamura,et al.  Methods to Segment Hard Inclusions in Soft Tissue During Autonomous Robotic Palpation , 2015, IEEE Transactions on Robotics.

[18]  J. Marescaux,et al.  Augmented-reality-assisted laparoscopic adrenalectomy. , 2004, JAMA.

[19]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[20]  V. Ozben,et al.  Robotic-Assisted Minimally Invasive Surgery , 2019, Springer International Publishing.

[21]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[22]  Chen Yang,et al.  Development of an inexpensive tri-axial force sensor for minimally invasive surgery , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Howie Choset,et al.  Estimating SE(3) elements using a dual quaternion based linear Kalman filter , 2016, Robotics: Science and Systems.

[24]  Howie Choset,et al.  Bingham Distribution-Based Linear Filter for Online Pose Estimation , 2017, Robotics: Science and Systems.

[25]  Nigel H. Lovell,et al.  A review of tactile sensing technologies with applications in biomedical engineering , 2012 .

[26]  Howie Choset,et al.  Trajectory-Optimized Sensing for Active Search of Tissue Abnormalities in Robotic Surgery , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Howie Choset,et al.  Complementary model update: A method for simultaneous registration and stiffness mapping in flexible environments , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Kaspar Althoefer,et al.  Rolling Mechanical Imaging for Tissue Abnormality Localization During Minimally Invasive Surgery , 2010, IEEE Transactions on Biomedical Engineering.

[29]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[30]  Russell H. Taylor,et al.  Iterative Most-Likely Point Registration (IMLP): A Robust Algorithm for Computing Optimal Shape Alignment , 2015, PloS one.

[31]  L.D. Seneviratne,et al.  State-of-the-Art in Force and Tactile Sensing for Minimally Invasive Surgery , 2008, IEEE Sensors Journal.

[32]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  Allison M. Okamura,et al.  Tissue property estimation and graphical display for teleoperated robot-assisted surgery , 2009, 2009 IEEE International Conference on Robotics and Automation.