Appearance learning for 3D tracking of robotic surgical tools

In this paper, we present an appearance learning approach which is used to detect and track surgical robotic tools in laparoscopic sequences. By training a robust visual feature descriptor on low-level landmark features, we build a framework for fusing robot kinematics and 3D visual observations to track surgical tools over long periods of time across various types of environment. We demonstrate 3D tracking on multiple types of tool (with different overall appearances) as well as multiple tools simultaneously. We present experimental results using the da Vinci® surgical robot using a combination of both ex-vivo and in-vivo environments.

[1]  David G. Stork,et al.  Pattern Classification , 1973 .

[2]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[3]  Yuan-Fang Wang,et al.  Image analysis for automated tracking in robot-assisted endoscopic surgery , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[4]  G. Hirzinger,et al.  Real-time visual servoing for laparoscopic surgery. Controlling robot motion with color image segmentation , 1997, IEEE Engineering in Medicine and Biology Magazine.

[5]  Gerd Hirzinger,et al.  Automatic tracking of laparoscopic instruments by color coding , 1997, CVRMed.

[6]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[8]  M. Mack,et al.  Minimally invasive and robotic surgery. , 2001, JAMA.

[9]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[10]  Xiaoli Zhang,et al.  Application of visual tracking for robot-assisted laparoscopic surgery , 2002, J. Field Robotics.

[11]  Min-Seok Kim,et al.  복강경 수술용 도구의 실시간 영상 추적 및 복강경 조종기의 지능형 제어 방법 = Real-time visual servoing for laparoscopic surgery , 2003 .

[12]  Luc Soler,et al.  Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing , 2003, IEEE Trans. Robotics Autom..

[13]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[14]  Florent Nageotte,et al.  Detection of grey regions in color images : application to the segmentation of a surgical instrument in robotized laparoscopy , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[15]  Xavier Pennec,et al.  A Riemannian Framework for Tensor Computing , 2005, International Journal of Computer Vision.

[16]  Pierre Graebling,et al.  Real-time segmentation of surgical instruments inside the abdominal cavity using a joint hue saturation color feature , 2005, Real Time Imaging.

[17]  Darius Burschka,et al.  DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability , 2005, MICCAI.

[18]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[19]  Darius Burschka,et al.  Navigating inner space: 3-D assistance for minimally invasive surgery , 2005, Robotics Auton. Syst..

[20]  Vincent Lepetit,et al.  Keypoint recognition using randomized trees , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  Fatih Murat Porikli,et al.  Region Covariance: A Fast Descriptor for Detection and Classification , 2006, ECCV.

[22]  Florent Nageotte,et al.  Segmentation and Guidance of Multiple Rigid Objects for Intra-operative Endoscopic Vision , 2006, WDV.

[23]  Philippe Cinquin,et al.  Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders , 2007, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[24]  Fatih Murat Porikli,et al.  Human Detection via Classification on Riemannian Manifolds , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[26]  Gerd Hirzinger,et al.  Motion Tracking for Minimally Invasive Robotic Surgery , 2008 .

[27]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.

[28]  Austin Reiter,et al.  An online learning approach to in-vivo tracking using synergistic features , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[29]  Kinematics Based Safety Operation Mechanism for Robotic Surgery extending the JHU SAW Framework , 2011, The MIDAS Journal.

[30]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[31]  Philippe Cinquin,et al.  3D Tracking of Laparoscopic Instruments Using Statistical and Geometric Modeling , 2011, MICCAI.

[32]  P. Allen,et al.  Marker-less Articulated Surgical Tool Detection , 2012 .

[33]  Austin Reiter,et al.  Feature Classification for Tracking Articulated Surgical Tools , 2012, MICCAI.

[34]  Austin Reiter,et al.  Learning features on robotic surgical tools , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[35]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[36]  Floris Ernst,et al.  Medical Robotics , 2015, Springer International Publishing.