Surface Mosaics of the Bladder Reconstructed From Endoscopic Video for Automated Surveillance

Flexible cystoscopy is frequently performed for recurrent bladder cancer surveillance, making it the most expensive cancer to treat over the patient's lifetime. An automated bladder surveillance system is being developed to robotically scan the bladder surface using an ultrathin and highly flexible endoscope. Such a system would allow cystoscopic procedures to be overseen by technical staff while urologists could review cystoscopic video postoperatively. In this paper, we demonstrate a method for reconstructing the surface of the whole bladder from endoscopic video using structure from motion. Video is acquired from a custom ultrathin and highly flexible endoscope that can retroflex to image the entire internal surface of the bladder. Selected frames are subsequently stitched into a mosaic and mapped to a reconstructed surface, creating a 3-D surface model of the bladder that can be expediently reviewed. Our software was tested on endoscopic video of an excised pig bladder. The resulting reconstruction possessed a projection error of 1.66 pixels on average and covered 99.6% of the bladder surface area.

[1]  Darius Burschka,et al.  Scale-invariant registration of monocular endoscopic images to CT-scans for sinus surgery , 2005, Medical Image Anal..

[2]  Gregory D. Hager,et al.  Direct global adjustment methods for endoscopic mosaicking , 2009, Medical Imaging.

[3]  A. Jemal,et al.  Cancer Statistics, 2010 , 2010, CA: a cancer journal for clinicians.

[4]  Sangtae Park,et al.  Development of an Automated Steering Mechanism for Bladder Urothelium Surveillance. , 2009, Journal of medical devices.

[5]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[6]  Christian Daul,et al.  Mosaicing of Bladder Endoscopic Image Sequences: Distortion Calibration and Registration Algorithm , 2008, IEEE Transactions on Biomedical Engineering.

[7]  Guang-Zhong Yang,et al.  i-BRUSH: A Gaze-Contingent Virtual Paintbrush for Dense 3D Reconstruction in Robotic Assisted Surgery , 2009, MICCAI.

[8]  Eric J. Seibel,et al.  Automated Cystoscopic Surveillance System with Endoscopic Image Mosaicing , 2010 .

[9]  Charles Soussen,et al.  A novel 3D surface construction approach: Application to three-dimensional endoscopic data , 2010, 2010 IEEE International Conference on Image Processing.

[10]  Steffen Leonhardt,et al.  Real-time image composition of bladder mosaics in fluorescence endoscopy , 2011, Computer Science - Research and Development.

[11]  Steven M. Seitz,et al.  Rectified Surface Mosaics , 2007, ICCV.

[12]  Baoxin Li,et al.  Synthesis of stereoscopic views from monocular endoscopic videos , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[13]  D. Wolf,et al.  Combination of panoramic and fluorescence endoscopic images to obtain tumor spatial distribution information useful for bladder cancer detection , 2007, European Conference on Biomedical Optics.

[14]  Edward H. Adelson,et al.  A multiresolution spline with application to image mosaics , 1983, TOGS.

[15]  Eric J. Seibel,et al.  Constructing spherical panoramas of a bladder phantom from endoscopic video using bundle adjustment , 2011, Medical Imaging.

[16]  Charles Soussen,et al.  From 2D towards 3D cartography of hollow organs , 2010, 2010 7th International Conference on Electrical Engineering Computing Science and Automatic Control.

[17]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[18]  D. Paulus,et al.  Feature-based real-time endoscopic mosaicking , 2009, 2009 Proceedings of 6th International Symposium on Image and Signal Processing and Analysis.

[19]  Richard S. Johnston,et al.  A full-color scanning fiber endoscope , 2006, SPIE BiOS.

[20]  Alexander Behrens.  Creating Panoramic Images for Bladder Fluorescence Endoscopy , 2008 .

[21]  Gregory D. Hager,et al.  Real-Time Endoscopic Mosaicking , 2006, MICCAI.

[22]  Til Aach,et al.  2D and 3D visualization methods of endoscopic panoramic bladder images , 2011, Medical Imaging.

[23]  G. Wahba.  Spline models for observational data , 1990 .

[24]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[25]  Chia-Hsiang Wu,et al.  Three-Dimensional Modeling From Endoscopic Video Using Geometric Constraints Via Feature Positioning , 2007, IEEE Transactions on Biomedical Engineering.

[26]  Richard Szeliski,et al.  Creating full view panoramic image mosaics and environment maps , 1997, SIGGRAPH.

[27]  Christian Daul,et al.  Fast construction of panoramic images for cystoscopic exploration , 2010, Comput. Medical Imaging Graph..

[28]  Eric J. Seibel,et al.  1389 TOWARDS SEMI AUTOMATED CYSTOSCOPIC EXAMINATION OF THE BLADDER: MERGING SCANNING FIBER ENDOSCOPE TECHNOLOGY WITH IMAGE PANORAMIC STITCHING SOFTWARE , 2011 .

[29]  Guang-Zhong Yang,et al.  Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery , 2006, MICCAI.

[30]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[31]  Carsten Rother,et al.  Linear Multi View Reconstruction and Camera Recovery Using a Reference Plane , 2002, International Journal of Computer Vision.

[32]  Michael Figl,et al.  Reconstruction of a 3D surface from video that is robust to missing data and outliers: Application to minimally invasive surgery using stereo and mono endoscopes , 2012, Medical Image Anal..

[33]  Guang-Zhong Yang,et al.  Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[34]  Michael B. Kimmey,et al.  Tethered Capsule Endoscopy, A Low-Cost and High-Performance Alternative Technology for the Screening of Esophageal Cancer and Barrett's Esophagus , 2008, IEEE Transactions on Biomedical Engineering.

[35]  T. Aach,et al.  Local and global panoramic imaging for fluorescence bladder endoscopy , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[36]  Nassir Navab,et al.  A Global Approach for Automatic Fibroscopic Video Mosaicing in Minimally Invasive Diagnosis , 2008, MICCAI.

[37]  Timothy D. Soper,et al.  Scanning fiber endoscopy with highly flexible, 1 mm catheterscopes for wide‐field, full‐color imaging , 2010, Journal of biophotonics.

[38]  Sebastian Bodenstedt,et al.  Real-time surface reconstruction from stereo endoscopic images for intraoperative registration , 2011, Medical Imaging.

[39]  Max Q.-H. Meng,et al.  3D reconstruction of the WCE images by affine SIFT method , 2011, 2011 9th World Congress on Intelligent Control and Automation.

[40]  James G Bittner,et al.  Three-dimensional visualisation and articulating instrumentation: Impact on simulated laparoscopic tasks , 2008, Journal of Minimal Access Surgery.

[41]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[42]  Kazuhiro Araki,et al.  Usefulness of panoramic views for novice surgeons doing retroperitoneal laparoscopic nephrectomy , 2009, International journal of urology : official journal of the Japanese Urological Association.

[43]  Andrew J. Davison,et al.  Real-time simultaneous localisation and mapping with a single camera , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[44]  Long Quan,et al.  Linear N-Point Camera Pose Determination , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Matthew A. Brown,et al.  Automatic Panoramic Image Stitching using Invariant Features , 2007, International Journal of Computer Vision.