Toward Long-Term and Accurate Augmented-Reality for Monocular Endoscopic Videos

By overlaying preoperative radiological 3-D models onto the intraoperative laparoscopic video, augmented-reality (AR) displays promise to increase surgeons' visual awareness of high-risk surgical targets (e.g., the location of a tumor). Existing AR surgical systems lack in robustness and accuracy because of the many challenges in endoscopic imagery, such as frequent changes in illumination, rapid camera motions, prolonged organ occlusions, and tissue deformations. The frequent occurrence of these events can cause the loss of image (anchor) points, and thus, the loss of the AR display after a few frames. In this paper, we present the design of a new AR system that represents a first step toward long term and accurate augmented surgical display for monocular (calibrated and uncalibrated) endoscopic videos. Our system uses correspondence-search methods, and a new weighted sliding-window registration approach, to automatically and accurately recover the overlay by predicting the image locations of a high number of anchor points that were lost after a sudden image change. The effectiveness of the proposed system in maintaining a long term (over 2 min) and accurate (less than 1 mm) augmentation has been documented over a set of real partial-nephrectomy laparascopic videos.

[1]  Daniel Pizarro-Perez,et al.  Realtime Wide-Baseline Registration of the Uterus in Laparoscopic Videos Using Multiple Texture Maps , 2013, AE-CAI.

[2]  Stefano Soatto,et al.  Real-Time Feature Tracking and Outlier Rejection with Changes in Illumination , 2001, ICCV.

[3]  Guang-Zhong Yang,et al.  Intraoperative Ultrasound Guidance for Transanal Endoscopic Microsurgery , 2012, MICCAI.

[4]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[5]  H. Meinzer,et al.  Augmented reality visualization during laparoscopic radical prostatectomy. , 2011, Journal of endourology.

[6]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[7]  Danail Stoyanov,et al.  Surgical Vision , 2011, Annals of Biomedical Engineering.

[8]  Guang-Zhong Yang,et al.  Augmented Reality Image Guidance in Minimally Invasive Prostatectomy , 2010, Prostate Cancer Imaging.

[9]  Michael C. Yip,et al.  Real-Time Methods for Long-Term Tissue Feature Tracking in Endoscopic Scenes , 2012, IPCAI.

[10]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[11]  Lena Maier-Hein,et al.  Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Gian Luca Mariottini,et al.  A Fast and Accurate Feature-Matching Algorithm for Minimally-Invasive Endoscopic Images , 2013, IEEE Transactions on Medical Imaging.

[14]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Didier Stricker,et al.  Online camera pose estimation in partially known and dynamic scenes , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[16]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[17]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[18]  Osamu Ukimura,et al.  Image-fusion, augmented reality, and predictive surgical navigation. , 2009, The Urologic clinics of North America.

[19]  Guang-Zhong Yang,et al.  Probabilistic Tracking of Affine-Invariant Anisotropic Regions , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Guang-Zhong Yang,et al.  Dynamic Guidance for Robotic Surgery Using Image-Constrained Biomechanical Models , 2010, MICCAI.

[21]  Guang-Zhong Yang,et al.  Motion Compensated SLAM for Image Guided Surgery , 2010, MICCAI.

[22]  Suya You,et al.  Natural feature tracking for extendible robust augmented realities , 1998 .

[23]  Russell H. Taylor,et al.  Video to CT Registration for Image Overlay on Solid Organs , 2008 .

[24]  Philippe Poignet,et al.  Towards robust 3D visual tracking for motion compensation in beating heart surgery , 2011, Medical Image Anal..

[25]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Gian Luca Mariottini,et al.  Real-Time Feature Matching for the Accurate Recovery of Augmented-Reality Display in Laparoscopic Videos , 2012, AE-CAI.

[27]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[28]  Stefano Soatto,et al.  Real-time feature tracking and outlier rejection with changes in illumination , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[29]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[30]  Eigil Samset,et al.  Augmented Reality for Minimally Invasive Surgery: Overview and Some Recent Advances , 2010 .

[31]  Paolo Cignoni,et al.  MeshLab: an Open-Source 3D Mesh Processing System , 2008, ERCIM News.

[32]  Gian Luca Mariottini,et al.  Hierarchical Multi-Affine (HMA) algorithm for fast and accurate feature matching in minimally-invasive surgical images , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[33]  William E. Higgins,et al.  3D CT-Video Fusion for Image-Guided Bronchoscopy , 2008, Comput. Medical Imaging Graph..

[34]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[35]  Gregory Hager,et al.  Vision-based navigation in image-guided interventions. , 2011, Annual review of biomedical engineering.

[36]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.