Inside-Out Instrument Tracking for Surgical Navigation in Augmented Reality

Surgical navigation requires tracking of instruments with respect to the patient. Conventionally, tracking is done with stationary cameras, and the navigation information is displayed on a stationary display. In contrast, an augmented reality (AR) headset can superimpose surgical navigation information directly in the surgeon’s view. However, AR needs to track the headset, the instruments and the patient, often by relying on stationary infrastructure. We show that 6DOF tracking can be obtained without any stationary, external system by purely utilizing the on-board stereo cameras of a HoloLens 2 to track the same retro-reflective marker spheres used by current optical navigation systems. Our implementation is based on two tracking pipelines complementing each other, one using conventional stereo vision techniques, the other relying on a single-constraint-at-a-time extended Kalman filter. In a technical evaluation of our tracking approach, we show that clinically relevant accuracy of 1.70 mm/1.11° and real-time performance is achievable. We further describe an example application of our system for untethered end-to-end surgical navigation.

[1]  Fabrizio Cutolo,et al.  Proof of Concept: Wearable Augmented Reality Video See-Through Display for Neuro-Endoscopy , 2018, AVR.

[2]  Kostas Daniilidis,et al.  Hand-Eye Calibration Using Dual Quaternions , 1999, Int. J. Robotics Res..

[3]  Sun-Joo Jang,et al.  An augmented reality system for image guidance of transcatheter procedures for structural heart disease , 2019, PloS one.

[4]  Ralf Dörner,et al.  Accuracy in optical tracking with fiducial markers: an accuracy function for ARToolKit , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[5]  Long Chen,et al.  Recent Developments and Future Challenges in Medical Mixed Reality , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[6]  S. Roner,et al.  Augmented reality for base plate component placement in reverse total shoulder arthroplasty: a feasibility study , 2020, Archives of Orthopaedic and Trauma Surgery.

[7]  Rafeef Garbi,et al.  Augmented reality visualisation for orthopaedic surgical guidance with pre‐ and intra‐operative multimodal image data fusion , 2018, Healthcare Technology Letters.

[8]  Ronald Azuma,et al.  Improving static and dynamic registration in an optical see-through HMD , 1994, SIGGRAPH.

[9]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[10]  S. Vijayakumar,et al.  RigidFusion: Robot Localisation and Mapping in Environments With Large Dynamic Rigid Objects , 2020, IEEE Robotics and Automation Letters.

[11]  Alexandru Tupan,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[12]  Harald Ganster,et al.  Hybrid Tracking for Augmented Reality , 2000 .

[13]  Francisco José Madrid-Cuevas,et al.  Automatic generation and detection of highly reliable fiducial markers under occlusion , 2014, Pattern Recognit..

[14]  I. Kuhlemann,et al.  Abstracts DGAV , 2018, Innovative Surgical Sciences.

[15]  Marc Pollefeys,et al.  HoloLens 2 Research Mode as a Tool for Computer Vision Research , 2020, ArXiv.

[16]  K. Ahrar,et al.  Percutaneous biopsy of head and neck lesions with CT guidance: various approaches and relevant anatomic and technical considerations. , 2007, Radiographics : a review publication of the Radiological Society of North America, Inc.

[17]  John J. Leonard,et al.  Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age , 2016, IEEE Transactions on Robotics.

[18]  Martin Weinmann,et al.  Evaluation of HoloLens Tracking and Depth Sensing for Indoor Mapping Applications , 2020, Sensors.

[19]  David J. Hawkes,et al.  Augmented reality needle ablation guidance tool for irreversible electroporation in the pancreas , 2018, Medical Imaging.

[20]  John J. Leonard,et al.  Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age , 2016, IEEE Transactions on Robotics.

[21]  Namgyu Kim,et al.  POSTRACK: a low cost real-time motion tracking system for VR application , 2001, Proceedings Seventh International Conference on Virtual Systems and Multimedia.

[22]  Bernd Schwald.  A Tracking Algorithm for Rigid Point-Based Marker Models , 2005 .

[23]  Brian A. Hargreaves,et al.  Mixed-Reality Guidance for Brain Stimulation Treatment of Depression , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct).

[24]  Peter F. Sturm,et al.  Triangulation , 1997, Comput. Vis. Image Underst..

[25]  Antonio Pepe,et al.  Markerless Image-to-Face Registration for Untethered Augmented Reality in Head and Neck Surgery , 2019, MICCAI.

[26]  Qingfeng Li,et al.  A Novel Evaluation Model for a Mixed-Reality Surgical Navigation System: Where Microsoft HoloLens Meets the Operating Room , 2020, Surgical innovation.

[27]  Wolfram Burgard,et al.  A benchmark for the evaluation of RGB-D SLAM systems , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[28]  Greg Welch,et al.  The HiBall Tracker: high-performance wide-area tracking for virtual and augmented environments , 1999, VRST '99.

[29]  Sylvain Chagué,et al.  A hand‐eye calibration method for augmented reality applied to computer‐assisted orthopedic surgery , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[30]  Tianpei Yang,et al.  Augmented Reality Guided Respiratory Liver Tumors Punctures: A Preliminary Feasibility Study , 2019, SIGGRAPH Asia Technical Briefs.

[31]  Peter Kazanzides,et al.  An Interactive Mixed Reality Platform for Bedside Surgical Procedures , 2020, MICCAI.

[32]  Floris Ernst,et al.  Towards X-ray free endovascular interventions – using HoloLens for on-line holographic visualisation , 2017, Healthcare technology letters.

[33]  Klaus Dorfmüller,et al.  Robust tracking for augmented reality using retroreflective markers , 1999, Comput. Graph..

[34]  Ronald Azuma,et al.  Hybrid inertial and vision tracking for augmented reality registration , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[35]  Le Xie,et al.  A feasibility study of a new method to enhance the augmented reality navigation effect in mandibular angle split osteotomy. , 2019, Journal of cranio-maxillo-facial surgery : official publication of the European Association for Cranio-Maxillo-Facial Surgery.

[36]  Horst Bischof,et al.  Learning Depth Calibration of Time-of-Flight Cameras , 2015, BMVC.

[37]  Axel Pinz,et al.  A new optical tracking system for virtual and augmented reality applications , 2001, IMTC 2001. Proceedings of the 18th IEEE Instrumentation and Measurement Technology Conference. Rediscovering Measurement in the Age of Informatics (Cat. No.01CH 37188).

[38]  Peter Kazanzides,et al.  Prioritization and static error compensation for multi-camera collaborative tracking in augmented reality , 2017, 2017 IEEE Virtual Reality (VR).

[39]  M. Kraft,et al.  Role of ultrasound‐guided core‐needle biopsy in the assessment of head and neck lesions: A meta‐analysis and systematic review of the literature , 2012, Head & neck.

[40]  Jan-Michael Frahm,et al.  Variable baseline/resolution stereo , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  F. Mathis-Ullrich,et al.  Infrared marker tracking with the HoloLens for neurosurgical interventions , 2020 .

[42]  Florentin Liebmann,et al.  Augmented Reality Navigation for Spinal Pedicle Screw Instrumentation using Intraoperative 3D Imaging. , 2020, The spine journal : official journal of the North American Spine Society.

[43]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[44]  Atsushi Konno,et al.  Marker-based motion tracking using Microsoft Kinect , 2018, SyRoCo.

[45]  Nassir Navab,et al.  Closing the Calibration Loop: An Inside-out-tracking Paradigm for Augmented Reality in Orthopedic Surgery , 2018, MICCAI.

[46]  Tong Xi,et al.  Toward Holographic-Guided Surgery , 2018, Surgical innovation.

[47]  Xiaoyi Jiang,et al.  Fast and accurate online calibration of optical see-through head-mounted display for AR-based surgical navigation using Microsoft HoloLens , 2020, International Journal of Computer Assisted Radiology and Surgery.

[48]  Greg Welch,et al.  SCAAT: incremental tracking with incomplete information , 1997, SIGGRAPH.

[49]  Peter Kazanzides,et al.  ARAMIS: Augmented Reality Assistance for Minimally Invasive Surgery Using a Head-Mounted Display , 2019, MICCAI.

[50]  Kangsoo Kim,et al.  Revisiting Trends in Augmented Reality Research: A Review of the 2nd Decade of ISMAR (2008–2017) , 2018, IEEE Transactions on Visualization and Computer Graphics.

[51]  Jean-Yves Bouguet,et al.  Camera calibration toolbox for matlab , 2001 .

[52]  Marc Pollefeys,et al.  A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[53]  Klaus H. Hinrichs,et al.  Generating optimized marker-based rigid bodies for optical tracking systems , 2007, VISAPP.

[54]  Christina Gsaxner,et al.  Facial model collection for medical augmented reality in oncologic cranio-maxillofacial surgery , 2019, Scientific Data.

[55]  I. Kuhlemann,et al.  Navigation and visualisation with HoloLens in endovascular aortic repair , 2018, Innovative surgical sciences.

[56]  G. Strauß,et al.  Image‐guided navigation: the surgeon's perspective on performance consequences and human factors issues , 2009, The international journal of medical robotics + computer assisted surgery : MRCAS.

[57]  Davide Scaramuzza,et al.  Pedicle screw navigation using surface digitization on the Microsoft HoloLens , 2019, International Journal of Computer Assisted Radiology and Surgery.

[58]  J. Kumar,et al.  Recent Development of Augmented Reality in Surgery: A Review , 2017, Journal of healthcare engineering.

[59]  Michael Brand,et al.  Accuracy of Marker Tracking on an Optical See-Through Head Mounted Display , 2020 .

[60]  S. Umeyama,et al.  Least-Squares Estimation of Transformation Parameters Between Two Point Patterns , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[62]  Nassir Navab,et al.  Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[63]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[64]  David Black,et al.  Auditory support for resection guidance in navigated liver surgery , 2013, The international journal of medical robotics + computer assisted surgery : MRCAS.

[65]  A. Sutton,et al.  A systematic review and meta-analysis of endovascular repair (EVAR) for ruptured abdominal aortic aneurysm. , 2008, European journal of vascular and endovascular surgery : the official journal of the European Society for Vascular Surgery.

[66]  Kai Lawonn,et al.  Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion , 2019, IEEE Transactions on Visualization and Computer Graphics.

[67]  Peter Kazanzides,et al.  ARssist: augmented reality on a head-mounted display for the first assistant in robotic surgery , 2018, Healthcare technology letters.

[68]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[69]  H. Kuhn.  The Hungarian method for the assignment problem , 1955 .

[70]  Florian Kral,et al.  Comparison of optical and electromagnetic tracking for navigated lateral skull base surgery , 2013, The international journal of medical robotics + computer assisted surgery : MRCAS.

[71]  U. Mezger,et al.  Navigation in surgery , 2013, Langenbeck's Archives of Surgery.

[72]  J. Wang,et al.  Navigation-guided core needle biopsy for skull base and parapharyngeal lesions: a five-year experience. , 2020, International journal of oral and maxillofacial surgery.