Real-time geometry-aware augmented reality in minimally invasive surgery

The potential of augmented reality (AR) technology to assist minimally invasive surgery (MIS) lies in its computational performance and accuracy in dealing with challenging MIS scenes. Even with the latest hardware and software technologies, achieving both real-time and accurate augmented information overlay in MIS is still a formidable task. In this Letter, the authors present a novel real-time AR framework for MIS that achieves interactive geometric aware AR in endoscopic surgery with stereo views. The authors’ framework tracks the movement of the endoscopic camera and simultaneously reconstructs a dense geometric mesh of the MIS scene. The movement of the camera is predicted by minimising the re-projection error to achieve a fast tracking performance, while the three-dimensional mesh is incrementally built by a dense zero mean normalised cross-correlation stereo-matching method to improve the accuracy of the surface reconstruction. The proposed system does not require any prior template or pre-operative scan and can infer the geometric information intra-operatively in real time. With the geometric information available, the proposed AR framework is able to interactively add annotations, localisation of tumours and vessels, and measurement labelling with greater precision and accuracy compared with the state-of-the-art approaches.

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  Zoltan-Csaba Marton,et al.  On fast surface reconstruction methods for large and noisy point clouds , 2009, 2009 IEEE International Conference on Robotics and Automation.

[3]  Danail Stoyanov,et al.  Robust surface tracking combining features, intensity and illumination compensation , 2015, International Journal of Computer Assisted Radiology and Surgery.

[4]  Danail Stoyanov,et al.  Robust Real-Time Visual Odometry for Stereo Endoscopy Using Dense Quadrifocal Tracking , 2014, IPCAI.

[5]  Guang-Zhong Yang,et al.  Soft tissue tracking for minimally invasive surgery: learning local deformation online. , 2008, Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention.

[6]  Adrien Bartoli,et al.  Tracking by Detection for Interactive Image Augmentation in Laparoscopy , 2012, WBIR.

[7]  Alexandre Hostettler,et al.  ORBSLAM-Based Endoscope Tracking and 3D Reconstruction , 2016, CARE@MICCAI.

[8]  Guang-Zhong Yang,et al.  Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[9]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[10]  Nazim Haouchine,et al.  Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery , 2015, IEEE Transactions on Visualization and Computer Graphics.

[11]  Adrien Bartoli,et al.  Use of augmented reality in laparoscopic gynecology to visualize myomas. , 2017, Fertility and sterility.

[12]  Kai-Che Liu,et al.  Stereoscopic visualization of laparoscope image using depth information from 3D model , 2014, Comput. Methods Programs Biomed..

[13]  Guang-Zhong Yang,et al.  Motion Compensated SLAM for Image Guided Surgery , 2010, MICCAI.

[14]  J. M. M. Montiel,et al.  Visual SLAM for Handheld Monocular Endoscope , 2014, IEEE Transactions on Medical Imaging.

[15]  Danail Stoyanov,et al.  Real-Time Dense Stereo Reconstruction Using Convex Optimisation with a Cost-Volume for Image-Guided Robotic Surgery , 2013, MICCAI.

[16]  Guang-Zhong Yang,et al.  Three-Dimensional Tissue Deformation Recovery and Tracking , 2010, IEEE Signal Processing Magazine.

[17]  Yu Sun,et al.  Simultaneous Tracking, 3D Reconstruction and Deforming Point Detection for Stereoscope Guided Surgery , 2013, AE-CAI.

[18]  Guang-Zhong Yang,et al.  Dense Surface Reconstruction for Enhanced Navigation in MIS , 2011, MICCAI.

[19]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[20]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[21]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[22]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[23]  Nazim Haouchine,et al.  Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[24]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[25]  Guang-Zhong Yang,et al.  Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery , 2006, MICCAI.

[26]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[28]  Zoltan-Csaba Marton,et al.  On Fast Surface Reconstruction Methods for Large and Noisy Datasets , 2009, IEEE International Conference on Robotics and Automation.

[29]  W. Kratzer,et al.  Factors Affecting Liver Size , 2003, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[30]  Nazim Haouchine,et al.  Patient-Specific Biomechanical Modeling for Guidance During Minimally-Invasive Hepatic Surgery , 2015, Annals of Biomedical Engineering.

[31]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[32]  Helder Araújo,et al.  A Non-Rigid Map Fusion-Based RGB-Depth SLAM Method for Endoscopic Capsule Robots , 2017, ArXiv.