3-D Pose Estimation of Articulated Instruments in Robotic Minimally Invasive Surgery

Estimating the 3-D pose of instruments is an important part of robotic minimally invasive surgery for automation of basic procedures as well as providing safety features, such as virtual fixtures. Image-based methods of 3-D pose estimation provide a non-invasive low cost solution compared with methods that incorporate external tracking systems. In this paper, we extend our recent work in estimating rigid 3-D pose with silhouette and optical flow-based features to incorporate the articulated degrees-of-freedom (DOFs) of robotic instruments within a gradient-based optimization framework. Validation of the technique is provided with a calibrated ex-vivo study from the da Vinci Research Kit (DVRK) robotic system, where we perform quantitative analysis on the errors each DOF of our tracker. Additionally, we perform several detailed comparisons with recently published techniques that combine visual methods with kinematic data acquired from the joint encoders. Our experiments demonstrate that our method is competitively accurate while relying solely on image data.

[1]  Anthony J. Yezzi,et al.  Robust 3D Pose Estimation and Efficient 2D Region-Based Segmentation from a 3D Shape Prior , 2008, ECCV.

[2]  Bodo Rosenhahn,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence Combined Region-and Motion-based 3d Tracking of Rigid and Articulated Objects , 2022 .

[3]  Sébastien Ourselin,et al.  Image Based Surgical Instrument Pose Estimation with Multi-class Labelling and Optical Flow , 2015, MICCAI.

[4]  Bernt Schiele,et al.  Detecting Surgical Tools by Modelling Local Appearance and Global Shape , 2015, IEEE Transactions on Medical Imaging.

[5]  Russell H. Taylor,et al.  Unified Detection and Tracking in Retinal Microsurgery , 2011, MICCAI.

[6]  Andrew W. Fitzgibbon,et al.  What Shape Are Dolphins? Building 3D Morphable Models from 2D Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[7]  J. Dankelman,et al.  Haptics in minimally invasive surgery – a review , 2008, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[8]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[9]  Cordelia Schmid,et al.  MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild , 2016, NIPS.

[10]  Nassir Navab,et al.  Concurrent Segmentation and Localization for Tracking of Surgical Instruments , 2017, MICCAI.

[11]  V. Ozben,et al.  Robotic-Assisted Minimally Invasive Surgery , 2019, Springer International Publishing.

[12]  Danail Stoyanov,et al.  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, IEEE Transactions on Medical Imaging.

[13]  Ian D. Reid,et al.  PWP3D: Real-time Segmentation and Tracking of 3D Objects , 2009, BMVC.

[14]  Pascal Fua,et al.  Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery , 2017, MICCAI.

[15]  Daniel Cremers,et al.  Dynamical statistical shape priors for level set-based tracking , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Austin Reiter,et al.  Appearance learning for 3D tracking of robotic surgical tools , 2014, Int. J. Robotics Res..

[17]  Perry Y. Li,et al.  ACHIEVING DEXTEROUS MANIPULATION FOR MINIMALLY INVASIVE SURGICAL ROBOTS THROUGH THE USE OF HYDRAULICS , 2012 .

[18]  Hans-Peter Seidel,et al.  Robust Pose Estimation with 3D Textured Models , 2006, PSIVT.

[19]  Ian D. Reid,et al.  Nonlinear shape manifolds as shape priors in level set segmentation and tracking , 2011, CVPR 2011.

[20]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[21]  Antonio Criminisi,et al.  Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning and Semi-Supervised Learning , 2012, Found. Trends Comput. Graph. Vis..

[22]  Bodo Rosenhahn,et al.  Region-Based Pose Tracking , 2007, IbPRIA.

[23]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[24]  Rüdiger Dillmann,et al.  Recognition of risk situations based on endoscopic instrument tracking and knowledge based situation modeling , 2008, SPIE Medical Imaging.

[25]  Rachid Deriche,et al.  A Review of Statistical Approaches to Level Set Segmentation: Integrating Color, Texture, Motion and Shape , 2007, International Journal of Computer Vision.

[26]  Adrien Bartoli,et al.  Structure-from-motion using lines: Representation, triangulation, and bundle adjustment , 2005, Comput. Vis. Image Underst..

[27]  J. Denavit,et al.  A kinematic notation for lower pair mechanisms based on matrices , 1955 .

[28]  Blake Hannaford,et al.  Roboscope: A flexible and bendable surgical robot for single portal Minimally Invasive Surgery , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Stefanie Speidel,et al.  Tracking of Instruments in Minimally Invasive Surgery for Surgical Skill Analysis , 2006, MIAR.

[30]  Andrew W. Fitzgibbon,et al.  Scene Coordinate Regression Forests for Camera Relocalization in RGB-D Images , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Guang-Zhong Yang,et al.  Real-Time 3D Tracking of Articulated Tools for Robotic Surgery , 2016, MICCAI.

[32]  Nassir Navab,et al.  Surgical Tool Tracking and Pose Estimation in Retinal Microsurgery , 2015, MICCAI.

[33]  Michael D. Naish,et al.  On constrained manipulation in robotics-assisted minimally invasive surgery , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[34]  Sébastien Ourselin,et al.  2D-3D Pose Tracking of Rigid Instruments in Minimally Invasive Surgery , 2014, IPCAI.

[35]  Peter Kontschieder,et al.  Structured class-labels in random forests for semantic image labelling , 2011, 2011 International Conference on Computer Vision.

[36]  Pascal Fua,et al.  Fast Part-Based Classification for Instrument Detection in Minimally Invasive Surgery , 2014, MICCAI.

[37]  Tony F. Chan,et al.  Level set based shape prior segmentation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[39]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[40]  J Dankelman,et al.  Systems for tracking minimally invasive surgical instruments , 2007, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[41]  P. Allen,et al.  Articulated Surgical Tool Detection Using Virtually-Rendered Templates , 2012 .

[42]  Xiaowei Zhou,et al.  Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.

[44]  Bodo Rosenhahn,et al.  Three-Dimensional Shape Knowledge for Joint Image Segmentation and Pose Estimation , 2005, DAGM-Symposium.