Hand-Eye Calibration of Surgical Instrument for Robotic Surgery Using Interactive Manipulation

Conventional robot hand-eye calibration methods are impractical for localizing robotic instruments in minimally-invasive surgeries under intra-corporeal workspace after pre-operative set-up. In this letter, we present a new approach to autonomously calibrate a robotic instrument relative to a monocular camera without recognizing calibration objects or salient features. The algorithm leverages interactive manipulation (IM) of the instrument for tracking its rigid-body motion behavior subject to the remote center-of-motion constraint. An adaptive controller is proposed to regulate the IM-induced instrument trajectory, using visual feedback, within a 3D plane which is observable from both the robot base and the camera. The eye-to-hand orientation and position are then computed via a dual-stage process allowing parameter estimation in low-dimensional spaces. The method does not require the exact knowledge of instrument model or large-scale data collection. Results from simulations and experiments on the da Vinci Research Kit are demonstrated via a laparoscopy resembled set-up using the proposed framework.

[1]  A. Lundberg,et al.  A new method for estimating the axis of rotation and the center of rotation. , 1999, Journal of biomechanics.

[2]  Abed Malti,et al.  Robust hand-eye calibration for computer aided medical endoscopy , 2010, 2010 IEEE International Conference on Robotics and Automation.

[3]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Alexandre Krupa,et al.  Pose Estimation and Feature Tracking for Robot Assisted Surgery with Medical Imaging , 2007 .

[5]  Oliver Brock,et al.  Interactive Perception: Leveraging Action in Perception and Perception in Action , 2016, IEEE Transactions on Robotics.

[6]  Pascal Fua,et al.  Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery , 2017, MICCAI.

[7]  Ève Coste-Manière,et al.  Flexible Calibration of Actuated Stereoscopic Endoscope for Overlay in Robot Assisted Surgery , 2002, MICCAI.

[8]  Danail Stoyanov,et al.  Hand-Eye Calibration With a Remote Centre of Motion , 2019, IEEE Robotics and Automation Letters.

[9]  Jason J. Corso,et al.  Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection , 2017, IEEE Transactions on Medical Imaging.

[10]  Danail Stoyanov,et al.  Adjoint Transformation Algorithm for Hand–Eye Calibration with Applications in Robotic Assisted Surgery , 2018, Annals of Biomedical Engineering.

[11]  Austin Reiter,et al.  Appearance learning for 3D tracking of robotic surgical tools , 2014, Int. J. Robotics Res..

[12]  Jiadong Shi,et al.  Robust image-based computation of the 3D position of RCM instruments and its application to image-guided manipulation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[13]  Bruno Siciliano,et al.  A V-REP Simulator for the da Vinci Research Kit Robotic Platform , 2018, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob).

[14]  Yun-Hui Liu,et al.  Dual-Arm Robotic Needle Insertion With Active Tissue Deformation for Autonomous Suturing , 2019, IEEE Robotics and Automation Letters.

[15]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[16]  Guang-Zhong Yang,et al.  A computationally efficient method for hand–eye calibration , 2017, International Journal of Computer Assisted Radiology and Surgery.

[17]  Christophe Doignon,et al.  A Degenerate Conic-Based Method for a Direct Fitting and 3-D Pose of Cylinders with a Single Perspective View , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[18]  Philippe Cinquin,et al.  3D Tracking of Laparoscopic Instruments Using Statistical and Geometric Modeling , 2011, MICCAI.

[19]  Danail Stoyanov,et al.  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, IEEE Transactions on Medical Imaging.

[20]  Allison M. Okamura,et al.  Methods to Segment Hard Inclusions in Soft Tissue During Autonomous Robotic Palpation , 2015, IEEE Transactions on Robotics.

[21]  Kostas Daniilidis,et al.  Hand-Eye Calibration Using Dual Quaternions , 1999, Int. J. Robotics Res..

[22]  Tommaso Falcone,et al.  Atlas of Single-Port, Laparoscopic, and Robotic Surgery , 2014 .

[23]  D. Stoyanov,et al.  3-D Pose Estimation of Articulated Instruments in Robotic Minimally Invasive Surgery , 2018, IEEE Transactions on Medical Imaging.

[24]  Roger Y. Tsai,et al.  A new technique for fully autonomous and efficient 3D robotics hand/eye calibration , 1988, IEEE Trans. Robotics Autom..

[25]  Philippe Cinquin,et al.  Automatic Localization of Laparoscopic Instruments for the Visual Servoing of an Endoscopic Camera Holder , 2006, MICCAI.

[26]  Oliver Brock,et al.  Manipulating articulated objects with interactive perception , 2008, 2008 IEEE International Conference on Robotics and Automation.

[27]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Ryan S. Decker,et al.  Supervised autonomous robotic soft tissue surgery , 2016, Science Translational Medicine.

[29]  Sébastien Ourselin,et al.  Image Based Surgical Instrument Pose Estimation with Multi-class Labelling and Optical Flow , 2015, MICCAI.

[30]  Ran Hao,et al.  Vision-Based Surgical Tool Pose Estimation for the da Vinci® Robotic Surgical System , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[31]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[32]  Paolo Fiorini,et al.  Medical Robotics and Computer-Integrated Surgery , 2008, 2008 32nd Annual IEEE International Computer Software and Applications Conference.

[33]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[34]  Nassir Navab,et al.  Concurrent Segmentation and Localization for Tracking of Surgical Instruments , 2017, MICCAI.

[35]  Danail Stoyanov,et al.  Hand-eye calibration for robotic assisted minimally invasive surgery without a calibration object , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Guang-Zhong Yang,et al.  Real-Time 3D Tracking of Articulated Tools for Robotic Surgery , 2016, MICCAI.

[37]  Heinrich Niemann,et al.  Robust Hand-Eye Calibration of an Endoscopic Surgery Robot Using Dual Quaternions , 2003, DAGM-Symposium.

[38]  J. Andrew Bagnell,et al.  Interactive segmentation, tracking, and kinematic modeling of unknown 3D articulated objects , 2013, 2013 IEEE International Conference on Robotics and Automation.

[39]  Kenneth Y. Goldberg,et al.  Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[40]  Yunhui Liu,et al.  Vision-Based Calibration of Dual RCM-Based Robot Arms in Human-Robot Collaborative Minimally Invasive Surgery , 2018, IEEE Robotics and Automation Letters.

[41]  Nassir Navab,et al.  Surgical Tool Tracking and Pose Estimation in Retinal Microsurgery , 2015, MICCAI.

[42]  Michael C. Yip,et al.  Robot Autonomy for Surgery , 2017, The Encyclopedia of Medical Robotics.