A Holographic Augmented Reality Interface for Visualizing of MRI Data and Planning of Neurosurgical Procedures

The recent introduction of wireless head-mounted displays (HMD) promises to enhance 3D image visualization by immersing the user into 3D morphology. This work introduces a prototype holographic augmented reality (HAR) interface for the 3D visualization of magnetic resonance imaging (MRI) data for the purpose of planning neurosurgical procedures. The computational platform generates a HAR scene that fuses pre-operative MRI sets, segmented anatomical structures, and a tubular tool for planning an access path to the targeted pathology. The operator can manipulate the presented images and segmented structures and perform path-planning using voice and gestures. On-the-fly, the software uses defined forbidden-regions to prevent the operator from harming vital structures. In silico studies using the platform with a HoloLens HMD assessed its functionality and the computational load and memory for different tasks. A preliminary qualitative evaluation revealed that holographic visualization of high-resolution 3D MRI data offers an intuitive and interactive perspective of the complex brain vasculature and anatomical structures. This initial work suggests that immersive experiences may be an unparalleled tool for planning neurosurgical procedures.

[1]  Nikolaos V Tsekos,et al.  Performance of interventions with manipulator-driven real-time MR guidance: implementation and initial in vitro tests. , 2007, Magnetic resonance imaging.

[2]  J. Peters,et al.  Closing the sensorimotor loop: haptic feedback facilitates decoding of motor imagery , 2011, Journal of neural engineering.

[3]  Nikolaos V. Tsekos,et al.  A Prototype Holographic Augmented Reality Interface for Image-Guided Prostate Cancer Interventions , 2018, VCBM@MICCAI.

[4]  Nikolaos V. Tsekos,et al.  Automated Segmentation and 4D Reconstruction of the Heart Left Ventricle from CINE MRI , 2019, 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering (BIBE).

[5]  Ozgur Kocaturk,et al.  Visualization of active devices and automatic slice repositioning (“SnapTo”) for MRI‐guided interventions , 2010, Magnetic resonance in medicine.

[6]  Nikolaos V. Tsekos,et al.  Towards MRI-guided and actuated tetherless milli-robots: Preoperative planning and modeling of control , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[7]  Kostas E. Bekris,et al.  Visual and force-feedback guidance for robot-assisted interventions in the beating heart with real-time MRI , 2012, 2012 IEEE International Conference on Robotics and Automation.

[8]  Fabrizio Cutolo,et al.  Robust and Accurate Algorithm for Wearable Stereoscopic Augmented Reality with Three Indistinguishable Markers , 2016 .

[9]  E R McVeigh,et al.  Techniques for fast stereoscopic MRI , 2001, Magnetic resonance in medicine.

[10]  Julien Abi-Nahed,et al.  End‐user evaluation of software‐generated intervention planning environment for transrectal magnetic resonance‐guided prostate biopsies , 2020, The international journal of medical robotics + computer assisted surgery : MRCAS.

[11]  Nikolaos V. Tsekos,et al.  A modular and scalable computational framework for interactive immersion into imaging data with a holographic augmented reality interface , 2021, Comput. Methods Programs Biomed..

[12]  D. Louis Collins,et al.  Towards Augmented Reality Guided Craniotomy Planning in Tumour Resections , 2016, MIAR.

[13]  K. Weimer,et al.  Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room , 2017, Plastic and reconstructive surgery.

[14]  Milan Sonka,et al.  3D Slicer as an image computing platform for the Quantitative Imaging Network. , 2012, Magnetic resonance imaging.

[15]  Nikolaos V. Tsekos,et al.  A Platform Integrating Acquisition, Reconstruction, Visualization, and Manipulator Control Modules for MRI-Guided Interventions , 2018, Journal of Digital Imaging.

[16]  Peter Kazanzides,et al.  Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2D-display , 2017, International Journal of Computer Assisted Radiology and Surgery.

[17]  R. Kikinis,et al.  Integration of interventional MRI with computer‐assisted surgery , 2001, Journal of magnetic resonance imaging : JMRI.

[18]  R Kikinis,et al.  Surgical navigation in the open MRI. , 2003, Acta neurochirurgica. Supplement.

[19]  Klaus D. Tönnies,et al.  Segmentation of medical images using adaptive region growing , 2001, SPIE Medical Imaging.

[20]  Floris Ernst,et al.  Towards X-ray free endovascular interventions – using HoloLens for on-line holographic visualisation , 2017, Healthcare technology letters.

[21]  Khae Hawn Kim.  The Potential Application of Virtual, Augmented, and Mixed Reality in Neurourology , 2016, International neurourology journal.

[22]  R. Kikinis,et al.  Development and implementation of intraoperative magnetic resonance imaging and its neurosurgical applications. , 1997, Neurosurgery.

[23]  Zhigang Deng,et al.  A Framework for Integrating Real-Time MRI With Robot Control: Application to Simulated Transapical Cardiac Interventions , 2013, IEEE Transactions on Biomedical Engineering.

[24]  Nikolaos V. Tsekos,et al.  Evaluation of Interventional Planning Software Features for MR-guided Transrectal Prostate Biopsies , 2020, 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE).

[25]  Samuel J Lin,et al.  Discussion: Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room. , 2017, Plastic and reconstructive surgery.

[26]  Enrico Gobbetti,et al.  Interactive virtual angioscopy , 1998, Proceedings Visualization '98 (Cat. No.98CB36276).

[27]  H. Gudbjartsson,et al.  The rician distribution of noisy mri data , 1995, Magnetic resonance in medicine.

[28]  Jeffrey L Duerk,et al.  The catheter-driven MRI scanner: a new approach to intravascular catheter tracking and imaging-parameter adjustment for interventional MRI. , 2004, AJR. American journal of roentgenology.

[29]  E. McVeigh,et al.  Real-time, Interactive MRI for cardiovascular interventions. , 2005, Academic radiology.

[30]  Jonathan M Sorger,et al.  Real-time volume rendered MRI for interventional guidance. , 2002, Journal of cardiovascular magnetic resonance : official journal of the Society for Cardiovascular Magnetic Resonance.

[31]  D. Stoianovici,et al.  Magnetic resonance imaging compatible robotic system for fully automated brachytherapy seed placement. , 2006, Urology.

[32]  Nikolaos V. Tsekos,et al.  Preliminary Evaluation of Robotic Transrectal Biopsy System on an Interventional Planning Software , 2019, 2019 IEEE 19th International Conference on Bioinformatics and Bioengineering (BIBE).

[33]  Nikolaos V. Tsekos,et al.  Holographic Interface for three-dimensional Visualization of MRI on HoloLens: A Prototype Platform for MRI Guided Neurosurgeries , 2017, 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE).

[34]  Zhigang Deng,et al.  GPU-Accelerated Interactive Visualization and Planning of Neurosurgical Interventions , 2014, IEEE Computer Graphics and Applications.