Assistive visual tools for surgery

This thesis describes efforts towards the development of computational vision tools to assist physicians during surgical procedures. Surgical robotics has come a long way towards improving patient care and enhancing the abilities of the surgeon. It was originally introduced to address the limitations of manual laparoscopy, whereby long rigid tools are being used to perform complicated surgical procedures. However, as we continue to reduce the invasiveness of this hardware and the tools becomes more complicated (e.g., flexible, articulated), intelligent algorithms are necessary to assist the surgeon in both controlling the surgical robot and making it smarter. Tool tracking algorithms are investigated to extract the locations of the surgical tools in both 2-dimensions (for visual servoing of a motorized camera system) and 3-dimensions (for situational awareness, increased patient safety, and virtual measuring capabilities). This work is shown in real surgical scenarios, such as manual human laparoscopies as well as robotic procedures using Intuitive Surgical's da Vinci® robot. Additionally, newer procedural paradigms which reduce the invasiveness of surgical access (e.g., Single-Port Access (SPA), Natural Orifice Translumenal Endoscopic Surgery (NOTES)) require more flexible and dexterous hardware. Vision is used to measure the state of continuum ("snake") robots in order to provide feedback to control systems and be able to perform fully-automated, closed-loop actions (e.g., suturing, biopsies, etc). This work is performed on the Insertable Robotic Effector Platform (IREP), which is a dexterous, 2-arm snake manipulator with a motorized stereo camera system designed for SPA/NOTES procedures.

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  Fatih Murat Porikli,et al.  Human Detection via Classification on Riemannian Manifolds , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  M. Mack,et al.  Minimally invasive and robotic surgery. , 2001, JAMA.

[4]  Philippe Cinquin,et al.  Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders , 2007, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[5]  Pierre E. Dupont,et al.  Design and Control of Concentric-Tube Robots , 2010, IEEE Transactions on Robotics.

[6]  John Kenneth Salisbury,et al.  Vision based 3-D shape sensing of flexible manipulators , 2008, 2008 IEEE International Conference on Robotics and Automation.

[7]  Philippe Cinquin,et al.  3D Tracking of Laparoscopic Instruments Using Statistical and Geometric Modeling , 2011, MICCAI.

[8]  Ian D. Walker,et al.  Extension versus Bending for Continuum Robots , 2006, ICINCO.

[9]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[10]  Vincent Lepetit,et al.  Keypoint recognition using randomized trees , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Yanxi Liu,et al.  Online selection of discriminative tracking features , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Hiroshi Murase,et al.  Visual learning and recognition of 3-d objects from appearance , 2005, International Journal of Computer Vision.

[13]  Bin Yao,et al.  Control of cable actuated devices using smooth backlash inverse , 2010, 2010 IEEE International Conference on Robotics and Automation.

[14]  J. Romanelli,et al.  Single-port laparoscopic surgery: an overview , 2009, Surgical Endoscopy.

[15]  Austin Reiter,et al.  A learning algorithm for visual pose estimation of continuum robots , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[17]  Yuan-Fang Wang,et al.  Image analysis for automated tracking in robot-assisted endoscopic surgery , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[18]  Robert J. Webster,et al.  Design and Kinematic Modeling of Constant Curvature Continuum Robots: A Review , 2010, Int. J. Robotics Res..

[19]  Austin Reiter,et al.  Feature Classification for Tracking Articulated Surgical Tools , 2012, MICCAI.

[20]  Philip H. S. Torr,et al.  The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix , 1997, International Journal of Computer Vision.

[21]  Peter Alken,et al.  Hans Christian Jacobaeus: Inventor of human laparoscopy and thoracoscopy. , 2006, Journal of endourology.

[22]  Xiaoli Zhang,et al.  Application of visual tracking for robot-assisted laparoscopic surgery , 2002, J. Field Robotics.

[23]  Andrew D. Back,et al.  Radial Basis Functions , 2001 .

[24]  D Marr,et al.  Theory of edge detection , 1979, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[25]  Robert T. Collins,et al.  On-the-fly Object Modeling while Tracking , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Shigeo Hirose,et al.  Biologically Inspired Robots: Snake-Like Locomotors and Manipulators , 1993 .

[27]  D. Caleb Rucker,et al.  Visual sensing of continuum robot shape using self-organizing maps , 2010, 2010 IEEE International Conference on Robotics and Automation.

[28]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.

[29]  Dock Bumpers,et al.  Volume 2 , 2005, Proceedings of the Ninth International Conference on Computer Supported Cooperative Work in Design, 2005..

[30]  Ian D. Walker,et al.  Vision based shape estimation for continuum robots , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[31]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[32]  Darius Burschka,et al.  Navigating inner space: 3-D assistance for minimally invasive surgery , 2005, Robotics Auton. Syst..

[33]  Yihong Gong,et al.  Discriminative learning of visual words for 3D human pose estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Vincent Lepetit,et al.  Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes , 2011, 2011 International Conference on Computer Vision.

[35]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[36]  Robert D. Howe,et al.  Position Control of Motion Compensation Cardiac Catheters , 2011, IEEE Transactions on Robotics.

[37]  John Kenneth Salisbury,et al.  Mechanics Modeling of Tendon-Driven Continuum Manipulators , 2008, IEEE Transactions on Robotics.

[38]  Gerd Hirzinger,et al.  Automatic tracking of laparoscopic instruments by color coding , 1997, CVRMed.

[39]  N. Simaan,et al.  Learning-based configuration estimation of a multi-segment continuum robot , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[40]  Austin Reiter,et al.  Learning features on robotic surgical tools , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[41]  Gerd Hirzinger,et al.  Motion Tracking for Minimally Invasive Robotic Surgery , 2008 .

[42]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[43]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[44]  P. Allen,et al.  Marker-less Articulated Surgical Tool Detection , 2012 .

[45]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[46]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[47]  Ryutarou Ohbuchi,et al.  Salient local visual features for shape-based 3D model retrieval , 2008, 2008 IEEE International Conference on Shape Modeling and Applications.

[48]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[49]  Bernd Neumann,et al.  Optical flow , 1986, Workshop on Motion.

[50]  Russell H. Taylor,et al.  A dexterous system for laryngeal surgery , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[51]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[52]  Trevor Darrell,et al.  Fast pose estimation with parameter-sensitive hashing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[53]  A. Reiter,et al.  Lessons learned using the insertable robotic effector platform (IREP) for single port access surgery , 2013, Journal of Robotic Surgery.

[54]  Robert D. Howe,et al.  Design and control of motion compensation cardiac catheters , 2010, 2010 IEEE International Conference on Robotics and Automation.

[55]  Robert J. Webster,et al.  Mechanics of Precurved-Tube Continuum Robots , 2009, IEEE Transactions on Robotics.

[56]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[57]  Peter K. Allen,et al.  System design of an Insertable Robotic Effector Platform for Single Port Access (SPA) Surgery , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[58]  Nathan A. Wood,et al.  Robotic natural orifice translumenal endoscopic surgery , 2008, 2008 IEEE International Conference on Robotics and Automation.

[59]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[60]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.