Simultaneous augmented and virtual reality for surgical navigation

We use a passive articulated arm to track a calibrated end-effector mounted video camera. In real time, we can superimpose the live video view with the synchronized graphical view of CT-derived segmented object(s) of interest within a phantom skull (augmented reality (AR)) and provide the trajectory of the end-effector (translated to the focal point) in orthogonal image data scans and 3D models (VR). Augmented reality generation is a natural extension for the surgeon because it does both the 2D to 3D transformation and projects the views directly onto the patient view. However, there are distinct advantages for also having a VR (image guided surgery) view of the tools trajectory. Both AR and VR visualization have advantages and disadvantages depending on the stage of the surgery and surgeons should have the option to select. In this paper, we provide the software design and the network communication details of a multi-user, on-demand, near real-time simultaneous AR/VR system for surgical guidance.

[1]  Warren Robinett,et al.  Synthetic Experience:A Proposed Taxonomy , 1992, Presence: Teleoperators & Virtual Environments.

[2]  Paramate Horkaew,et al.  Optimal Deformable Surface Models for 3D Medical Image Analysis , 2003, IPMI.

[3]  Mohammad-Reza Siadat,et al.  Design, implementation and accuracy of a prototype for medical augmented reality , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[4]  Gábor Székely,et al.  Enhancing human-computer interaction in medical segmentation , 2003, Proc. IEEE.

[5]  J. L. Herring,et al.  Surface registration for use in interactive, image-guided liver surgery. , 2000 .

[6]  H Iseki,et al.  Augmented reality visualization system for intravascular neurosurgery. , 1998, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[7]  Yoshikazu Nakajima,et al.  Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization , 1998, IEEE Transactions on Medical Imaging.

[8]  Ulrich Neumann,et al.  Multiring Fiducial Systems for Scalable Fiducial-Tracking Augmented Reality , 2001, Presence: Teleoperators & Virtual Environments.

[9]  M AlcañizRaya,et al.  Mixed reality for neurosurgery: a novel prototype. , 2003 .

[10]  W. Eric L. Grimson,et al.  Coupled Multi-shape Model and Mutual Information for Medical Image Segmentation , 2003, IPMI.

[11]  Frank Sauer,et al.  An Augmented Reality system for MRI-guided needle biopsies. , 2003, Studies in health technology and informatics.

[12]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[13]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[14]  H Iseki,et al.  New Possibilities for Stereotaxis , 2001, Stereotactic and Functional Neurosurgery.

[15]  Takeo Kanade,et al.  An Image Overlay System for Medical Data Visualization , 1998, MICCAI.

[16]  Wolfgang Freysinger,et al.  Interactive telepresence and augmented reality in ENT surgery: interventional video tomography , 1997, CVRMed.

[17]  Wolfgang Broll,et al.  Interface with angels: the future of VIR and AR interfaces , 2001 .

[18]  P. Jannin,et al.  A data fusion environment for multimodal and multi-informational neuronavigation. , 2000, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[19]  Naoki Suzuki,et al.  A robotic surgery system (da Vinci) with image guided function--system architecture and cholecystectomy application. , 2003, Studies in health technology and informatics.

[20]  Andreas Pommert.  Medicine Meets Virtual Reality , 1997 .

[21]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[22]  Weihong Guo,et al.  using Prior Shape and Points in Medical Image Segmentation , 2003, EMMCVPR.

[23]  Tomas Lozano-Perez,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1996 .

[24]  Pau-Choo Chung,et al.  Identifying multiple abdominal organs from CT image series using a multimodule contextual neural network and spatial fuzzy rules , 2003, IEEE Transactions on Information Technology in Biomedicine.

[25]  Ioannis A. Kakadiaris,et al.  Tracking Methods for Medical Augmented Reality , 2001, MICCAI.

[26]  David J. Hawkes,et al.  AcouStick: A Tracked A-Mode Ultrasonography System for Registration in Image-Guided Surgery , 1999, MICCAI.

[27]  H E Cline,et al.  3D reconstruction of the brain from magnetic resonance images using a connectivity algorithm. , 1987, Magnetic Resonance Imaging.

[28]  R Darin Ellis,et al.  Development and human factors analysis of neuronavigation vs. augmented reality. , 2004, Studies in health technology and informatics.