Efficiently Calibrating Cable-Driven Surgical Robots With RGBD Sensing, Temporal Windowing, and Linear and Recurrent Neural Network Compensation

Automation of surgical subtasks using cabledriven robotic surgical assistants (RSAs) such as Intuitive Surgical’s da Vinci Research Kit (dVRK) is challenging due to imprecision in control from cable-related effects such as backlash, stretch, and hysteresis. We propose a novel approach to efficiently calibrate a dVRK by placing a 3D printed fiducial coordinate frame on the arm and end-effector that is tracked using RGBD sensing. To measure the coupling effects between joints and history-dependent effects, we analyze data from sampled trajectories and consider 13 modeling approaches using LSTM recurrent neural networks and linear models with varying temporal window length to provide corrective feedback. With the proposed method, data collection takes 31 minutes to produce 1800 samples and model training takes less than a minute. Results suggest that the resulting model can reduce the mean tracking error of the physical robot from 2.96 mm to 0.65 mm on a test set of reference trajectories. We evaluate the model by executing open-loop trajectories of the FLS peg transfer surgeon training task. Results suggest that the best approach increases success rate from 39.4 % to 96.7 % comparable to the performance of an expert surgical resident. Supplementary material, including 3Dprintable models, is available at https://sites.google. com/berkeley.edu/surgical-calibration

[1]  Jacob Rosen,et al.  Autonomous Operation in Surgical Robotics , 2015 .

[2]  Ji Ma,et al.  Autonomous suturing via surgical robot: An algorithm for optimal selection of needle diameter, shape, and path , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Priya Sundaresan,et al.  Automated Extraction of Surgical Needles from Tissue Phantoms , 2019, 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE).

[4]  Kirk A. Nichols,et al.  A Disposable Haptic Palpation Probe for Locating Subcutaneous Blood Vessels in Robot-Assisted Minimally Invasive Surgery , 2015 .

[5]  Pieter Abbeel,et al.  Learning accurate kinematic control of cable-driven surgical robots using data cleaning and Gaussian Process Regression , 2014, 2014 IEEE International Conference on Automation Science and Engineering (CASE).

[6]  Yun-Hui Liu,et al.  Dual-Arm Robotic Needle Insertion With Active Tissue Deformation for Autonomous Suturing , 2019, IEEE Robotics and Automation Letters.

[7]  Minho Hwang,et al.  Effect of backlash hysteresis of surgical tool bending joints on task performance in teleoperated flexible endoscopic robot , 2020, The international journal of medical robotics + computer assisted surgery : MRCAS.

[8]  Kenneth Y. Goldberg,et al.  Tumor localization using automated palpation with Gaussian Process Adaptive Sampling , 2016, 2016 IEEE International Conference on Automation Science and Engineering (CASE).

[9]  Pieter Abbeel,et al.  Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations , 2010, 2010 IEEE International Conference on Robotics and Automation.

[10]  Jin U. Kang,et al.  Autonomous Laparoscopic Robotic Suturing with a Novel Actuated Suturing Tool and 3D Endoscope , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[11]  Jacob Rosen,et al.  Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[12]  Katsu Yamane,et al.  Deep Imitation Learning of Sequential Fabric Smoothing Policies , 2019, ArXiv.

[13]  Blake Hannaford,et al.  Real-time Data Driven Precision Estimator for RAVEN-II Surgical Robot End Effector Position , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Blake Hannaford,et al.  Raven-II: An Open Platform for Surgical Robotics Research , 2013, IEEE Transactions on Biomedical Engineering.

[15]  Peter Kazanzides,et al.  An open-source research kit for the da Vinci® Surgical System , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Blake Hannaford,et al.  Measurement of the cable-pulley Coulomb and viscous friction for a cable-driven surgical robotic system , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Pieter Abbeel,et al.  Learning by observation for surgical subtasks: Multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Ankush Gupta,et al.  A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[19]  Michael C. Yip,et al.  Robot Autonomy for Surgery , 2017, The Encyclopedia of Medical Robotics.

[20]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[21]  Brijen Thananjeyan,et al.  Applying Depth-Sensing to Automated Surgical Manipulation with a da Vinci Robot , 2020, 2020 International Symposium on Medical Robotics (ISMR).

[22]  John F. Canny,et al.  Fast and Reliable Autonomous Surgical Debridement with Cable-Driven Robots Using a Two-Phase Calibration Procedure , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Elena De Momi,et al.  Automated Pick-Up of Suturing Needles for Robotic Surgical Assistance , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[24]  Blake Hannaford,et al.  Unscented Kalman Filter and 3D vision to improve cable driven surgical robot joint angle estimation , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[25]  Kenneth Y. Goldberg,et al.  Automating multi-throw multilateral surgical suturing with a mechanical needle guide and sequential convex optimization , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Pieter Abbeel,et al.  Autonomous multilateral debridement with the Raven surgical robot , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[27]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[28]  Brijen Thananjeyan,et al.  Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[30]  Minho Hwang,et al.  K‐FLEX: A flexible robotic platform for scar‐free endoscopic surgery , 2020, The international journal of medical robotics + computer assisted surgery : MRCAS.

[31]  Gaurav S. Sukhatme,et al.  Learning task error models for manipulation , 2013, 2013 IEEE International Conference on Robotics and Automation.

[32]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.