Integrated Medical Workflow for Augmented Reality Applications

Augmented Reality (AR) and Virtual Reality (VR) have the potential to improve surgical interventions and image-guided therapy by providing clinicians with interactive three-dimensional visualizations in all phases of treatment. However, experimental AR research applications typically focus on a single medical procedure and do not attempt to cover the whole clinical workflow from preoperative to intra-operative and post-operative support. In the scope of the EU-funded Marie Curie research project ARIS*ER (Augmented Reality in Surgery), we are investigating a more complete workflow support with AR technology. This paper discusses a component-oriented software architecture which specifically aims at addressing the needs of medical AR, in particular strategies for ensuring uniform application quality throughout the medical workflow. We discuss the management of off-line and on-line image acquisition methods and methods for consistent management of modeling, processing and visualization of medical datasets throughout the medical workflow. Finally, we present three different medical application examples which illustrate the use of the framework. CR Categories: D.2.6 [Programming Environments]: Graphical environments—Integrated environmentsInteractive environmentsProgrammer workbench H.5.1 [Multimedia Information Systems]: Artificial, augmented, and virtual realities— [I.3.7]: ThreeDimensional Graphics and Realism—Virtual reality

[1]  Luc Soler,et al.  An augmented reality system to guide radio‐frequency tumour ablation , 2005, Comput. Animat. Virtual Worlds.

[2]  N. Hata,et al.  An integrated visualization system for surgical planning and guidance using image fusion and an open MR , 2001, Journal of magnetic resonance imaging : JMRI.

[3]  Herbert Edelsbrunner,et al.  Geometry and Topology for Mesh Generation , 2001, Cambridge monographs on applied and computational mathematics.

[4]  Bernd Brügge,et al.  Design of a component-based augmented reality framework , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[5]  William J. Schroeder,et al.  The visualization toolkit user's guide : updated for version 4.0 , 1998 .

[6]  Alexander Bornik,et al.  Augmented-reality-based segmentation refinement , 2004, Medical Imaging: Image-Guided Procedures.

[7]  Milan Sonka,et al.  Virtual Liver Surgery Planning , 2008 .

[8]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[9]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[10]  W. Eric L. Grimson,et al.  An Integrated Visualization System for Surgical Planning and Guidance Using Image Fusion and Interventional Imaging , 1999, MICCAI.

[11]  Hervé Delingette,et al.  General Object Reconstruction Based on Simplex Meshes , 1999, International Journal of Computer Vision.

[12]  Blair MacIntyre,et al.  DART: the Designer's Augmented Reality Toolkit , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[13]  Milan Sonka,et al.  Computer-aided liver surgery planning: an augmented reality approach , 2003, SPIE Medical Imaging.

[14]  Horst Bischof,et al.  A Novel Robust Tube Detection Filter for 3D Centerline Extraction , 2005, SCIA.

[15]  Frank Sauer,et al.  An augmented reality system for MR image-guided needle biopsy: initial results in a swine model. , 2006, Radiology.

[16]  Dieter Schmalstieg,et al.  A Hybrid User Interface for Manipulation of Volumetric Medical Data , 2006, 3D User Interfaces (3DUI'06).

[17]  Georgios Sakas,et al.  Computer-aided surgery based on auto-stereoscopic augmented reality , 2004, Proceedings. Eighth International Conference on Information Visualisation, 2004. IV 2004..

[18]  M. Brian Blake,et al.  IGSTK: an open source software toolkit for image-guided surgery , 2006, Computer.

[19]  Dieter Schmalstieg,et al.  An open software architecture for virtual reality interaction , 2001, VRST '01.

[20]  Paul S. Strauss,et al.  An object-oriented 3D graphics toolkit , 1992, SIGGRAPH.

[21]  Henry Fuchs,et al.  Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms , 2002, Medical Image Anal..

[22]  Milan Sonka,et al.  Liver segment approximation in CT data for surgical resection planning , 2004, SPIE Medical Imaging.