Plenoptic cameras in surgical robotics: Calibration, registration, and evaluation

Three-dimensional sensing of changing surgical scenes would improve the function of surgical robots. This paper explores the requirements and utility of a new type of depth sensor, the plenoptic camera, for surgical robots. We present a metric calibration procedure for the plenoptic camera and the registration of its coordinate frame to the robot (hand-eye calibration). We also demonstrate the utility in robotic needle insertion and application of sutures in phantoms. The metric calibration accuracy is reported as 1.14 ± 0.80 mm for the plenoptic camera and 1.57 ± 0.90 mm for hand-eye calibration. The accuracy of needle insertion task is 1.79 ± 0.35 mm for the entire robotic system. Additionally, the accuracy of suture placement with the presented system is reported at 1.80 ± 0.43 mm. Finally, we report consistent suture spacing with only 0.11 mm standard deviation between inter-suture distances. The measured accuracy of less than 2 mm with consistent suture spacing is a promising result to provide repeatable leak-free suturing with a robotic tool and a plenoptic depth imager.

[1]  Axel Krieger,et al.  Performance evaluation and clinical applications of 3D plenoptic cameras , 2015, Commercial + Scientific Sensing and Imaging.

[2]  Frederik Wenz,et al.  Intrafraction motion of the prostate during an IMRT session: a fiducial-based 3D measurement with Cone-beam CT , 2008, Radiation oncology.

[3]  Andrzej Napieralski,et al.  Computer-aided approach to evaluation of burn wounds , 2011, Proceedings of the 18th International Conference Mixed Design of Integrated Circuits and Systems - MIXDES 2011.

[4]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[5]  Danail Stoyanov,et al.  A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[6]  Michel Dhome,et al.  Hand-eye calibration , 1997, Proceedings of the 1997 IEEE/RSJ International Conference on Intelligent Robot and Systems. Innovative Robotics for Real-World Applications. IROS '97.

[7]  Danail Stoyanov,et al.  Visual Odometry in Stereo Endoscopy by Using PEaRL to Handle Partial Scene Deformation , 2014, AE-CAI.

[8]  Edward H. Adelson,et al.  Single Lens Stereo with a Plenoptic Camera , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Axel Krieger,et al.  Smart Tissue Anastomosis Robot (STAR): Accuracy evaluation for supervisory suturing using near-infrared fluorescent markers , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[11]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[12]  Benjamin J Vakoc,et al.  Photometric stereo endoscopy , 2013, Journal of biomedical optics.

[13]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[14]  Steve Gallivan,et al.  Surgical Performance Measurement , 2002, Health care management science.

[15]  Karin Kunzi-Rapp,et al.  Indications for Optical Shape Measurements in Orthopaedics and Dermatology , 2002 .

[16]  M. Schurr,et al.  Precision in stitches: Radius Surgical System , 2007, Surgical Endoscopy.

[17]  Roger Y. Tsai,et al.  A new technique for fully autonomous and efficient 3D robotics hand/eye calibration , 1988, IEEE Trans. Robotics Autom..

[18]  D. Oleynikov,et al.  Real-time three-dimensional soft tissue reconstruction for laparoscopic surgery , 2012, Surgical Endoscopy.

[19]  Gerd Hirzinger,et al.  Optimal Hand-Eye Calibration , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[20]  Graham M. Treece,et al.  3D ultrasound measurement of large organ volume , 2001, Medical Image Anal..

[21]  Bastian Goldlücke,et al.  On the Calibration of Focused Plenoptic Cameras , 2013, Time-of-Flight and Depth Imaging.

[22]  Frederick H Epstein,et al.  Measurement of myocardial mechanics in mice before and after infarction using multislice displacement-encoded MRI with 3D motion encoding. , 2005, American journal of physiology. Heart and circulatory physiology.

[23]  Axel Krieger,et al.  Smart Tissue Anastomosis Robot (STAR): A Vision-Guided Robotics System for Laparoscopic Suturing , 2014, IEEE Transactions on Biomedical Engineering.

[24]  S. Seki,et al.  Accuracy of suture placement , 1987, The British journal of surgery.

[25]  Jeffrey A. Cadeddu,et al.  Endoscopic stereo reconstruction: A comparative study , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[26]  G. Schreiber,et al.  The Fast Research Interface for the KUKA Lightweight Robot , 2022 .

[27]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[28]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[29]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  L. Fajardo,et al.  Near-infrared optical imaging of the breast with model-based reconstruction. , 2002, Academic radiology.