Vision-Based Force Feedback Estimation for Robot-Assisted Surgery Using Instrument-Constrained Biomechanical Three-Dimensional Maps

We present a method for estimating visual and haptic force feedback on robotic surgical systems that currently do not include significant force feedback for the operator. Our approach permits to compute contact forces between instruments and tissues without additional sensors, relying only on endoscopic images acquired by a stereoscopic camera. Using an underlying biomechanical model built on-the-fly from the organ shape and by considering the surgical tool as boundary conditions acting on the surface of the model, contact force can be estimated at the tip of the tool. At the same time, these constraints generate stresses that permit to compose a new endoscopic image as visual feedback for the surgeon. The results are demonstrated on in vivo sequences of a human liver during robotic surgery, whereas quantitative validation is performed on a DejaVu and ex vivo experimentation with ground truth to show the advantage of our approach.

[1]  B. Aronov,et al.  Discrete and computational geometry : the Goodman-Pollack Festschrift , 2003 .

[2]  Alessandro Gasparetto,et al.  A Force-Torque Sensor for the Applications in Medical Robotics , 2005 .

[3]  Nazim Haouchine,et al.  Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[4]  David Levin,et al.  The approximation power of moving least-squares , 1998, Math. Comput..

[5]  D. Burschka,et al.  Dynamic Augmented Reality for Sensory Substitution in Robot-Assisted Surgical Systems , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[6]  Andrew Nealen,et al.  Physically Based Deformable Models in Computer Graphics , 2006, Comput. Graph. Forum.

[7]  Lena Maier-Hein,et al.  Physics-based shape matching for intraoperative image guidance. , 2014, Medical physics.

[8]  Guang-Zhong Yang,et al.  pq-space Based Non-Photorealistic Rendering for Augmented Reality , 2007, MICCAI.

[9]  A. Okamura.  Haptic feedback in robot-assisted minimally invasive surgery , 2009, Current opinion in urology.

[10]  Sonny Chan,et al.  High-fidelity haptic and visual rendering for patient-specific simulation of temporal bone surgery , 2016, Computer assisted surgery.

[11]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[12]  Russell H. Taylor,et al.  A miniature microsurgical instrument tip force sensor for enhanced force feedback during robot-assisted manipulation , 2003, IEEE Trans. Robotics Autom..

[13]  Allison M. Okamura,et al.  Methods for haptic feedback in teleoperated robot-assisted surgery , 2004 .

[14]  Tamal K. Dey,et al.  Detecting undersampling in surface reconstruction , 2001, SCG '01.

[15]  Nazim Haouchine,et al.  DejaVu: Intra-operative Simulation for Surgical Gesture Rehearsal , 2017, MICCAI.

[16]  Danail Stoyanov,et al.  Stereoscopic Scene Flow for Robotic Assisted Minimally Invasive Surgery , 2012, MICCAI.

[17]  Nazim Haouchine,et al.  Segmentation and labelling of intra-operative laparoscopic images using structure from point cloud , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[18]  Guang-Zhong Yang,et al.  Hand-Held Medical Robots , 2014, Annals of Biomedical Engineering.

[19]  Paolo Dario,et al.  Vision-based haptic feedback for capsule endoscopy navigation: a proof of concept , 2016 .

[20]  Sébastien Ourselin,et al.  2D-3D Pose Tracking of Rigid Instruments in Minimally Invasive Surgery , 2014, IPCAI.

[21]  D. Yuh,et al.  Effects of visual force feedback on robot-assisted surgical task performance. , 2008, The Journal of thoracic and cardiovascular surgery.

[22]  Septimiu E. Salcudean,et al.  Bimanual telerobotic surgery with asymmetric force feedback: A daVinci® surgical system implementation , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[23]  Nazim Haouchine,et al.  Patient-Specific Biomechanical Modeling for Guidance During Minimally-Invasive Hepatic Surgery , 2015, Annals of Biomedical Engineering.

[24]  Motoji Yamamoto,et al.  An edge-based computationally efficient formulation of Saint Venant-Kirchhoff tetrahedral finite elements , 2009, ACM Trans. Graph..