A Real-Time 3D Laparoscopic Imaging System: Design, Method, and Validation

Objective: This paper aims to propose a 3D laparoscopic imaging system that can realize dense 3D reconstruction in real time. Methods: Based on the active stereo technique which yields high-density, accurate and robust 3D reconstruction by combining structured light and stereo vision, we design a laparoscopic system consisting of two image feedback channels and one pattern projection channel. Remote high-speed image acquisition and pattern generation lay the foundation for the real-time dense 3D surface reconstruction and enable the miniaturization of the laparoscopic probe. To enhance the reconstruction efficiency and accuracy, we propose a novel active stereo method by which the dense 3D point cloud is obtained using only five patterns, while most existing multiple-shot structured light techniques require $\text{10--40}$ patterns. In our method, dual-frequency phase-shifting fringes are utilized to uniquely encode the pixels of the measured targets, and a dual-codeword matching scheme is developed to simplify the matching procedure and achieve high-precision reconstruction. Results: Compared with the existing structured light techniques, the proposed method shows better real-time efficiency and accuracy in both quantitative and qualitative ways. Ex-vivo experiments demonstrate the robustness of the proposed method to different biological organs and the effectiveness to lesions and deformations of the organs. Feasibility of the proposed system for real-time dense 3D reconstruction is verified in dynamic experiments. According to the experimental results, the system acquires 3D point clouds with a speed of 12 frames per second. Each frame contains more than 40,000 points, and the average errors tested on standard objects are less than 0.2 mm. Significance: This paper provides a new real-time dense 3D reconstruction method for 3D laparoscopic imaging. The established prototype system has shown good performance in reconstructing surface of biological tissues.

[1]  Guang-Zhong Yang,et al.  Autonomous scanning for endomicroscopic mosaicing and 3D fusion , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[2]  Naoki Suzuki,et al.  Laser-scan endoscope system for intraoperative geometry acquisition and surgical robot safety management , 2006, Medical Image Anal..

[3]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  Alexandre Hostettler,et al.  Live Tracking and Dense Reconstruction for Handheld Monocular Endoscopy , 2019, IEEE Transactions on Medical Imaging.

[5]  Marion Semmler,et al.  3D Reconstruction of Human Laryngeal Dynamics Based on Endoscopic High-Speed Recordings , 2016, IEEE Transactions on Medical Imaging.

[6]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[7]  Richard K. Martin,et al.  Human skin detection in the visible and near infrared. , 2015, Applied optics.

[8]  Joaquim Salvi,et al.  Pattern codification strategies in structured light systems , 2004, Pattern Recognit..

[9]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[10]  Ping Li,et al.  Image-Aligned Dynamic Liver Reconstruction Using Intra-Operative Field of Views for Minimal Invasive Surgery , 2019, IEEE Transactions on Biomedical Engineering.

[11]  Gang Xu,et al.  Epipolar Geometry in Stereo, Motion and Object Recognition , 1996, Computational Imaging and Vision.

[12]  Austin Reiter,et al.  Surgical Structured Light for 3D minimally invasive surgical imaging , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Qian Chen,et al.  Real-time 3-D shape measurement with composite phase-shifting fringes and multi-view system. , 2016, Optics express.

[14]  Danail Stoyanov,et al.  Stereoscopic Scene Flow for Robotic Assisted Minimally Invasive Surgery , 2012, MICCAI.

[15]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[16]  Guang-Zhong Yang,et al.  Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging , 2011, Biomedical optics express.

[17]  Philippe Poignet,et al.  Towards robust 3D visual tracking for motion compensation in beating heart surgery , 2011, Medical Image Anal..

[18]  Jun Wang,et al.  Dynamic Reconstruction of Deformable Soft-Tissue With Stereo Scope in Minimal Invasive Surgery , 2020, IEEE Robotics and Automation Letters.

[19]  Sam Van der Jeught,et al.  Real-time structured light profilometry: a review , 2016 .

[20]  Antonios Gasteratos,et al.  Review of Stereo Vision Algorithms: From Software to Hardware , 2008 .

[21]  Tanneguy Redarce,et al.  Binary pattern codification strategies in an active stereoscopic system based on flexible image guides , 2013, Photonics West - Micro and Nano Fabricated Electromechanical and Optical Components.

[22]  Benoit M. Dawant,et al.  Organ Surface Deformation Measurement and Analysis in Open Hepatic Surgery: Method and Preliminary Results From 12 Clinical Cases , 2011, IEEE Transactions on Biomedical Engineering.

[23]  Joaquim Salvi,et al.  A state of the art in structured light patterns for surface profilometry , 2010, Pattern Recognit..

[24]  Axel Wiegmann,et al.  Human face measurement by projecting bandlimited random patterns. , 2006, Optics express.

[25]  Zerui Wang,et al.  A Robust Data-Driven Approach for Online Learning and Manipulation of Unmodeled 3-D Heterogeneous Compliant Objects , 2018, IEEE Robotics and Automation Letters.

[26]  Kai Xu,et al.  A foldable stereo vision unit for Single Port Access Laparoscopy , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Jostein Thorstensen,et al.  Structured Light Imaging under Sunlight Conditions , 2018 .

[28]  Yang Hu,et al.  Dual‐modality endoscopic probe for tissue surface shape reconstruction and hyperspectral imaging enabled by deep neural networks , 2018, Medical Image Anal..

[29]  Hongen Liao,et al.  Fast image mapping of endoscopic image mosaics with three-dimensional ultrasound image for intrauterine fetal surgery , 2009, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[30]  Jason Geng,et al.  Review of 3-D Endoscopic Surface Imaging Techniques , 2014, IEEE Sensors Journal.

[31]  Anthony J. Durkin,et al.  Review of short-wave infrared spectroscopy and imaging methods for biological tissue characterization , 2015, Journal of biomedical optics.

[32]  Terry M. Peters,et al.  Real-time fusion of endoscopic views with dynamic 3-D cardiac images: a phantom study , 2005, IEEE Transactions on Medical Imaging.

[33]  Song Zhang,et al.  High-speed 3D shape measurement with structured light methods: A review , 2018, Optics and Lasers in Engineering.

[34]  Yunhui Liu,et al.  A 3D Laparoscopic Imaging System Based on Stereo-Photogrammetry with Random Patterns , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[35]  Adrien Bartoli,et al.  Towards Live Monocular 3D Laparoscopy Using Shading and Specularity Information , 2012, IPCAI.

[36]  Faisal Mahmood,et al.  Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy , 2017, Medical Image Anal..

[37]  Jay B. West,et al.  The distribution of target registration error in rigid-body point-based registration , 2001, IEEE Transactions on Medical Imaging.

[38]  Tokuo Tsuji,et al.  High-speed 3D image acquisition using coded structured light projection , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  Hongen Liao,et al.  Ultrasound Image-Guided Mapping of Endoscopic Views on a 3D Placenta Model: A Tracker-Less Approach , 2013, AE-CAI.

[40]  Zhiwei Xiong,et al.  Real-Time Scalable Depth Sensing With Hybrid Structured Light Illumination , 2014, IEEE Transactions on Image Processing.

[41]  Nicolas Andreff,et al.  Endoscopic Laser Surgery: Design, Modeling, and Control , 2017, IEEE/ASME Transactions on Mechatronics.

[42]  Faisal Mahmood,et al.  Deep learning-based depth estimation from a synthetic endoscopy image training set , 2018, Medical Imaging.

[43]  Li Cheng,et al.  Vision-Based Surgical Suture Looping Through Trajectory Planning for Wound Suturing , 2019, IEEE Transactions on Automation Science and Engineering.