Three-Dimensional Tissue Deformation Recovery and Tracking

Recent advances in surgical robotics have provided a platform for extending the current capabilities of minimally invasive surgery by incorporating both preoperative and intraoperative imaging data. In this tutorial article, we introduce techniques for in vivo three-dimensional (3-D) tissue deformation recovery and tracking based on laparoscopic or endoscopic images. These optically based techniques provide a unique opportunity for recovering surface deformation of the soft tissue without the need of additional instrumentation. They can therefore be easily incorporated into the existing surgical workflow. Technically, the problem formulation is challenging due to nonrigid deformation of the tissue and instrument interaction. Current approaches and future research directions in terms of intraoperative planning and adaptive surgical navigation are explained in detail.

[1]  Peter Burger,et al.  Differential algorithm for the determination of shape from shading using a point light source , 1992, Image Vis. Comput..

[2]  Danail Stoyanov,et al.  A practical approach towards accurate dense 3D depth recovery for robotic laparoscopic surgery , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[3]  Vincent Lepetit,et al.  Monocular Model-Based 3D Tracking of Rigid Objects: A Survey , 2005, Found. Trends Comput. Graph. Vis..

[4]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[5]  Chao Liu,et al.  Deformable motion tracking of the heart surface , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Allison M. Okamura,et al.  Modeling of Tool-Tissue Interactions for Computer-Based Surgical Simulation: A Literature Review , 2008, PRESENCE: Teleoperators and Virtual Environments.

[7]  Tobias Ortmaier,et al.  Motion estimation in beating heart surgery , 2005, IEEE Transactions on Biomedical Engineering.

[8]  Sebastian Thrun,et al.  Online simultaneous localization and mapping with detection and tracking of moving objects: theory and results from a ground vehicle in crowded urban areas , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[9]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[10]  Takayuki Okatani,et al.  Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center , 1997, Comput. Vis. Image Underst..

[11]  Wolfram Burgard,et al.  Probabilistic Robotics (Intelligent Robotics and Autonomous Agents) , 2005 .

[12]  Russell H. Taylor,et al.  Medical robotics in computer-integrated surgery , 2003, IEEE Trans. Robotics Autom..

[13]  Luc Soler,et al.  Beating heart tracking in robotic surgery using 500 Hz visual servoing, model predictive control and an adaptive observer , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[14]  Guang-Zhong Yang,et al.  Illumination position estimation for 3D soft-tissue reconstruction in robotic minimally invasive surgery , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Guang-Zhong Yang,et al.  A Probabilistic Framework for Tracking Deformable Soft Tissue in Minimally Invasive Surgery , 2007, MICCAI.

[16]  J. Triboulet,et al.  Texture Features Selection for Visual Servoing of the Beating Heart , 2006, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[17]  Olivier D. Faugeras,et al.  Shape from shading: a well-posed problem? , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[18]  Gerd Hirzinger,et al.  OPTICAL FLOW TO ANALYSE STABILISED IMAGES OF THE BEATING HEART , 2006 .

[19]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[20]  Philippe Poignet,et al.  Three-dimensional heart motion estimation using endoscopic monocular vision system: From artificial landmarks to texture analysis , 2007, Biomed. Signal Process. Control..

[21]  Philippe C. Cattin,et al.  Endoscopic navigation for minimally invasive suturing , 2008, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[22]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[23]  Michael Figl,et al.  Non-rigid Reconstruction of the Beating Heart Surface for Minimally Invasive Cardiac Surgery , 2009, MICCAI.

[24]  Guang-Zhong Yang,et al.  Dynamic view expansion for minimally invasive surgery using simultaneous localization and mapping , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[25]  M. Brand.  Morphable 3 D models from video , 2001 .

[26]  Cameron N. Riviere,et al.  Predicting respiratory motion for active canceling during percutaneous needle insertion , 2001, 2001 Conference Proceedings of the 23rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  Christian Daul,et al.  Mosaicing of Bladder Endoscopic Image Sequences: Distortion Calibration and Registration Algorithm , 2008, IEEE Transactions on Biomedical Engineering.

[28]  Darius Burschka,et al.  Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery , 2004, MICCAI.

[29]  Branislav Jaramaz,et al.  A Multi-Image Shape-from-Shading Framework for Near-Lighting Perspective Endoscopes , 2009, International Journal of Computer Vision.

[30]  Guang-Zhong Yang,et al.  Affine-invariant anisotropic detector for soft tissue tracking in minimally invasive surgery , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[31]  Philip H. S. Torr,et al.  The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix , 1997, International Journal of Computer Vision.

[32]  Michael Figl,et al.  3D Reconstruction of Internal Organ Surfaces for Minimal Invasive Surgery , 2007, MICCAI.

[33]  Castaneda V,et al.  Reconstructing the Esophagus Surface from Endoscopic Image Sequences , 2009 .

[34]  Jacques Gangloff,et al.  Toward Robotized Beating Heart TECABG: Assessment of the Heart Dynamics Using High-Speed Vision , 2005, MICCAI.

[35]  Guang-Zhong Yang,et al.  Simultaneous Stereoscope Localization and Soft-Tissue Mapping for Minimal Invasive Surgery , 2006, MICCAI.

[36]  V. Shayani,et al.  Determination of the learning curve of the AESOP robot , 1997, Surgical Endoscopy.

[37]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  H. Barlow.  Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[39]  Guang-Zhong Yang,et al.  Dense 3D Depth Recovery for Soft Tissue Deformation During Robotically Assisted Laparoscopic Surgery , 2004, MICCAI.

[40]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[41]  Tatsuo Igarashi,et al.  Computer‐based endoscopic image‐processing technology for endourology and laparoscopic surgery , 2009, International journal of urology : official journal of the Japanese Urological Association.

[42]  Cameron N. Riviere,et al.  Robotic Compensation of Biological Motion to Enhance Surgical Accuracy , 2006, Proceedings of the IEEE.

[43]  D. Wolf,et al.  Combination of panoramic and fluorescence endoscopic images to obtain tumor spatial distribution information useful for bladder cancer detection , 2007, European Conference on Biomedical Optics.

[44]  Naoki Suzuki,et al.  Laser-scan endoscope system for intraoperative geometry acquisition and surgical robot safety management , 2006, Medical Image Anal..

[45]  Guang-Zhong Yang,et al.  Gaze contingent articulated robot control for robot assisted minimally invasive surgery , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[46]  Yuan-Fang Wang,et al.  Toward automated model building from video in computer-assisted diagnoses in colonoscopy , 2007, SPIE Medical Imaging.

[47]  Guang-Zhong Yang,et al.  A stereoscopic fibroscope for camera motion and 3D depth recovery during Minimally Invasive Surgery , 2009, 2009 IEEE International Conference on Robotics and Automation.

[48]  Michael B. Kimmey,et al.  Tethered Capsule Endoscopy, A Low-Cost and High-Performance Alternative Technology for the Screening of Esophageal Cancer and Barrett's Esophagus , 2008, IEEE Transactions on Biomedical Engineering.

[49]  Armin Schneider,et al.  Time-of-Flight 3-D Endoscopy , 2009, MICCAI.

[50]  Luc Van Gool,et al.  Real-time range acquisition by adaptive structured light , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Guang-Zhong Yang,et al.  Gaze-Contingent Soft Tissue Deformation Tracking for Minimally Invasive Robotic Surgery , 2005, MICCAI.

[52]  Philippe Zanne,et al.  Physiological motion rejection in flexible endoscopy using visual servoing , 2008, 2008 IEEE International Conference on Robotics and Automation.

[53]  Guang-Zhong Yang,et al.  Soft tissue tracking for minimally invasive surgery: learning local deformation online. , 2008, Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention.

[54]  Luc Soler,et al.  Active filtering of physiological motion in robotized surgery using predictive control , 2005, IEEE Transactions on Robotics.

[55]  Guang-Zhong Yang,et al.  Belief Propagation for Depth Cue Fusion in Minimally Invasive Surgery , 2008, MICCAI.

[56]  Gregory D. Hager,et al.  Real-Time Endoscopic Mosaicking , 2006, MICCAI.

[57]  Guang-Zhong Yang,et al.  i-BRUSH: A Gaze-Contingent Virtual Paintbrush for Dense 3D Reconstruction in Robotic Assisted Surgery , 2009, MICCAI.

[58]  A. Park,et al.  A technique for real-time digital measurements in laparoscopic surgery , 2004, Surgical Endoscopy And Other Interventional Techniques.

[59]  T. Peters.  Image-guidance for surgical procedures , 2006, Physics in medicine and biology.

[60]  H. Haneishi,et al.  Profilometry of a gastrointestinal surface by an endoscope with laser beam projection. , 1994, Optics letters.

[61]  Lorenzo Torresani,et al.  Tracking and modeling non-rigid objects with rank constraints , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[62]  Gregory D. Hager,et al.  Stereo-Based Endoscopic Tracking of Cardiac Surface Deformation , 2004, MICCAI.

[63]  Tobias Ortmaier,et al.  Tracking local motion on the beating heart , 2002, SPIE Medical Imaging.

[64]  Yehezkel Yeshurun,et al.  Perspective shape-from-shading by fast marching , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[65]  Pierre Graebling,et al.  Robust Structured Light Coding for 3D Reconstruction , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[66]  Carlos Henrique Quartucci Forster,et al.  Towards 3D reconstruction of endoscope images using shape from shading , 2000, Proceedings 13th Brazilian Symposium on Computer Graphics and Image Processing (Cat. No.PR00878).

[67]  M. de Mathelin,et al.  Comparison of visual tracking algorithms on in vivo sequences for robot-assisted flexible endoscopic surgery , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[68]  Baoxin Li,et al.  Circular generalized cylinder fitting for 3D reconstruction in endoscopic imaging based on MRF , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[69]  Ève Coste-Manière,et al.  3D reconstruction of the operating field for image overlay in 3D-endoscopic surgery , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[70]  Alexander Behrens.  Creating Panoramic Images for Bladder Fluorescence Endoscopy , 2008 .

[71]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[72]  M. Mack,et al.  Minimally invasive and robotic surgery. , 2001, JAMA.

[73]  Steven M. Seitz,et al.  Rectified Surface Mosaics , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[74]  Chia-Hsiang Wu,et al.  Three-Dimensional Modeling From Endoscopic Video Using Geometric Constraints Via Feature Positioning , 2007, IEEE Transactions on Biomedical Engineering.

[75]  Nassir Navab,et al.  A Global Approach for Automatic Fibroscopic Video Mosaicing in Minimally Invasive Diagnosis , 2008, MICCAI.

[76]  Philippe Poignet,et al.  Motion prediction for tracking the beating heart , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[77]  Brian L. Davies,et al.  Active-Constraint Robotics for Surgery , 2006, Proceedings of the IEEE.

[78]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[79]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[80]  Ralph J Damiano.  Robotics in cardiac surgery: the Emperor's new clothes. , 2007, The Journal of thoracic and cardiovascular surgery.