Visual computing for medical diagnosis and treatment

Diagnostic algorithms and efficient visualization techniques are of major importance for preoperative decisions, intra-operative imaging and image-guided surgery. Complex diagnostic decisions are characterized by a high information flow and fast decisions, requiring efficient and intuitive presentation of complex medical data and precision in the visualization. For intra-operative medical treatment, the pre-operative visualization results of the diagnostic systems have to be transferred to the patient on the operation room table. Via augmented reality, additional information of the hidden regions can be displayed virtually. This state-of-the-art report summarizes visual computing algorithms for medical diagnosis and treatment. After starting with direct volume rendering and tagged volume rendering as general techniques for visualizing anatomical structures, we go into more detail by focusing on the visualization of tissue and vessel structures. Afterwards, algorithms and techniques that are used for medical treatment in the context of image-guided surgery, intra-operative imaging and augmented reality are discussed and reviewed.

[1]  James K Takayesu,et al.  A comprehensive medical simulation education curriculum for emergency medicine residents. , 2007, Annals of emergency medicine.

[2]  Anders Ynnerman,et al.  Uncertainty Visualization in Medical Volume Rendering , 2007 .

[3]  Max A. Viergever,et al.  A survey of medical image registration , 1998, Medical Image Anal..

[4]  Helmut Seibert,et al.  A flexible tracking concept applied to medical scenarios using an AR window , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[5]  Gordon Kindlmann,et al.  Superquadric tensor glyphs , 2004, VISSYM'04.

[6]  Marie-Paule Cani,et al.  Real-Time Simulation of Self-collisions for Virtual Intestinal Surgery , 2003, IS4TH.

[7]  Mathieu Desbrun,et al.  Dynamic real-time deformations using space & time adaptive sampling , 2001, SIGGRAPH.

[8]  Peter Hastreiter,et al.  High performance volume splatting for visualization of neurovascular data , 2005, VIS 05. IEEE Visualization, 2005..

[9]  Ken Masamune,et al.  Region-Growing Based Feature Extraction Algorithm for Tree-Like Objects , 1996, VBC.

[10]  Guido Brunnett,et al.  Geometric Modeling for Scientific Visualization , 2010 .

[11]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[12]  Simon Stegmaier,et al.  A simple and flexible volume rendering framework for graphics-hardware-based raycasting , 2005, Fourth International Workshop on Volume Graphics, 2005..

[13]  B. Tomandl,et al.  CT angiography of intracranial aneurysms: a focus on postprocessing. , 2004, Radiographics : a review publication of the Radiological Society of North America, Inc.

[14]  Dirk Bartz,et al.  The Role of Perception for Computer Graphics , 2008, Eurographics.

[15]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[16]  Joseph S. B. Mitchell,et al.  Efficient Collision Detection Using Bounding Volume Hierarchies of k-DOPs , 1998, IEEE Trans. Vis. Comput. Graph..

[17]  Gordon L. Kindlmann,et al.  Strategies for Direct Volume Rendering of Diffusion Tensor Fields , 2000, IEEE Trans. Vis. Comput. Graph..

[18]  Holger Regenbrecht,et al.  Detecting Dynamic Occlusion in front of Static Backgrounds for AR Scenes , 2003, IPT/EGVE.

[19]  Joe Michael Kniss,et al.  Interactive Texture-Based Volume Rendering for Large Data Sets , 2001, IEEE Computer Graphics and Applications.

[20]  George Baciu,et al.  Hardware-assisted self-collision for deformable surfaces , 2002, VRST '02.

[21]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[22]  David E. Breen,et al.  Interactive Occlusion and Automatic Object Placement for Augmented Reality , 1996, Comput. Graph. Forum.

[23]  Pierre Hellier,et al.  Level Set Methods in an EM Framework for Shape Classification and Estimation , 2004, International Conference on Medical Image Computing and Computer-Assisted Intervention.

[24]  David Hawkes,et al.  Surface transparency makes stereo overlays unpredictable: the implications for augmented reality. , 2003, Studies in health technology and informatics.

[25]  Bernhard Preim,et al.  Visualization of vasculature with convolution surfaces: method, validation and evaluation , 2005, IEEE Transactions on Medical Imaging.

[26]  Dirk Bartz,et al.  Occlusion handling for medical augmented reality using a volumetric phantom model , 2004, VRST '04.

[27]  Hans-Heino Ehricke,et al.  Visualization of vasculature from volume data , 1994, Comput. Graph..

[28]  P. Basser,et al.  MR diffusion tensor spectroscopy and imaging. , 1994, Biophysical journal.

[29]  Joe Michael Kniss,et al.  Interactive volume rendering using multi-dimensional transfer functions and direct manipulation widgets , 2001, Proceedings Visualization, 2001. VIS '01..

[30]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[31]  Wolfgang Birkfellner,et al.  The Varioscope AR - A Head-Monted Operating Microscope for Augmented Reality , 2000, MICCAI.

[32]  Thomas Lange,et al.  Augmenting Intraoperative 3D Ultrasound with Preoperative Models for Navigation in Liver Surgery , 2004, MICCAI.

[33]  M. Bauer,et al.  Interactive volume on standard PC graphics hardware using multi-textures and multi-stage rasterization , 2000, Workshop on Graphics Hardware.

[34]  Heinz-Otto Peitgen,et al.  Grid-based spectral fiber clustering , 2007, SPIE Medical Imaging.

[35]  Carl-Fredrik Westin,et al.  A Bayesian approach for stochastic white matter tractography , 2006, IEEE Transactions on Medical Imaging.

[36]  O Ganslandt,et al.  Preliminary experience in glioma surgery with intraoperative high-field MRI. , 2003, Acta neurochirurgica. Supplement.

[37]  Rüdiger Westermann,et al.  Acceleration techniques for GPU-based volume rendering , 2003, IEEE Visualization, 2003. VIS 2003..

[38]  Dirk Bartz,et al.  State-of-the-Art of the Role of Perception for Computer Graphics , 2008 .

[39]  Dinesh K. Pai,et al.  BD-tree: output-sensitive collision detection for reduced deformable models , 2004, ACM Trans. Graph..

[40]  Anna Vilanova,et al.  DTI visualization with streamsurfaces and evenly-spaced volume seeding , 2004, VISSYM'04.

[41]  Michael Bajura,et al.  Merging Virtual Objects with the Real World , 1992 .

[42]  Bernhard Preim,et al.  Automatic Transfer Function Specification for Visual Emphasis of Coronary Artery Plaque , 2010, Comput. Graph. Forum.

[43]  James J. Troy,et al.  Six degree-of-freedom haptic rendering using voxel sampling , 1999, SIGGRAPH.

[44]  Derek K Jones,et al.  Applications of diffusion‐weighted and diffusion tensor MRI to white matter diseases – a review , 2002, NMR in biomedicine.

[45]  R. vanLiere,et al.  Proceedings of Eurographics Symposium on Virtual Environments , 2008 .

[46]  Bernhard Preim,et al.  Geometrical and Structural Analysis of Vessel Systems in 3D Medical Image Datasets , 2004 .

[47]  Markus H. Gross,et al.  Signed distance transform using graphics hardware , 2003, IEEE Visualization, 2003. VIS 2003..

[48]  Dirk Bartz,et al.  Virtual Endoscopy in Research and Clinical Practice , 2005, Comput. Graph. Forum.

[49]  Thomas Czech,et al.  ADVANCED VIRTUAL ENDOSCOPY FOR ENDOSCOPIC TRANSSPHENOIDAL PITUITARY SURGERY , 2006, Neurosurgery.

[50]  Bernhard Preim,et al.  Visualization and interaction techniques for the exploration of vascular structures , 2001, Proceedings Visualization, 2001. VIS '01..

[51]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[52]  Juho Kannala,et al.  A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[53]  Nassir Navab,et al.  Automatic Patient Registration for Port Placement in Minimally Invasixe Endoscopic Surgery , 2005, MICCAI.

[54]  Stan Melax,et al.  Dynamic Plane Shifting BSP Traversal , 2000, Graphics Interface.

[55]  David J. Hawkes,et al.  Design and evaluation of a system for microscope-assisted guided interventions (MAGI) , 1999, IEEE Transactions on Medical Imaging.

[56]  Bernhard Preim,et al.  Intraoperative augmented reality for minimally invasive liver interventions , 2003, SPIE Medical Imaging.

[57]  R Kikinis,et al.  Interventional and intraoperative magnetic resonance imaging. , 2000, Annual review of biomedical engineering.

[58]  Thomas Lange,et al.  Vessel-Based Non-Rigid Registration of MR/CT and 3D Ultrasound for Navigation in Liver Surgery , 2003, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[59]  Andreas Kolb,et al.  Opacity Peeling for Direct Volume Rendering , 2006, Comput. Graph. Forum.

[60]  Chris R. Johnson,et al.  NHI-NSF Visualization Research Challenges Report , 2005 .

[61]  Markus Hadwiger,et al.  Real‐Time Ray‐Casting and Advanced Shading of Discrete Isosurfaces , 2005, Comput. Graph. Forum.

[62]  Klaus Mueller,et al.  GPU accelerated image aligned splatting , 2005, Fourth International Workshop on Volume Graphics, 2005..

[63]  Marie-Odile Berger.  Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[64]  Stefan Bruckner,et al.  Memory efficient acceleration structures and techniques for CPU-based volume raycasting of large data , 2004 .

[65]  Anders Ynnerman,et al.  Full Body Virtual Autopsies using a State-of-the-art Volume Rendering Pipeline , 2006, IEEE Transactions on Visualization and Computer Graphics.

[66]  Donald H. House,et al.  Grid With a View: Optimal Texturing for Perception of Layered Surface Shape , 2007, IEEE Transactions on Visualization and Computer Graphics.

[67]  Jules Bloomenthal,et al.  Convolution surfaces , 1991, SIGGRAPH.

[68]  Mary C. Whitton,et al.  Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies , 1996, SIGGRAPH.

[69]  Katja Bühler,et al.  Geometric Methods for Vessel Visualization and Quantification — A Survey , 2004 .

[70]  Susumu Mori,et al.  Fiber tracking: principles and strategies – a technical review , 2002, NMR in biomedicine.

[71]  Dinesh Manocha,et al.  Fast and reliable collision culling using graphics hardware , 2005, SIGGRAPH Courses.

[72]  Ronald N. Perry,et al.  Adaptively sampled distance fields: a general representation of shape for computer graphics , 2000, SIGGRAPH.

[73]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[74]  Yonggang Lu,et al.  Improved fiber tractography with Bayesian tensor regularization , 2006, NeuroImage.

[75]  Thomas Ertl,et al.  Smart Hardware-Accelerated Volume Rendering , 2003, VisSym.

[76]  Ron Kikinis,et al.  White Matter Tractography Using Sequential Importance Sampling , 2002 .

[77]  Penny Rheingans,et al.  NIH-NSF visualization research challenges report summary , 2006, IEEE Computer Graphics and Applications.

[78]  Nassir Navab,et al.  Merging visible and invisible: two Camera-Augmented Mobile C-arm (CAMC) applications , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[79]  Laurent Grisoni,et al.  An intestinal surgery simulator: real-time collision processing and visualization , 2004, IEEE Transactions on Visualization and Computer Graphics.

[80]  David H. Laidlaw,et al.  Visualizing Diffusion Tensor MR Images Using Streamtubes and Streamsurfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[81]  Frank Sauer,et al.  Augmented reality visualization of ultrasound images: system description, calibration, and features , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[82]  Christof Rezk-Salama,et al.  High-Level User Interfaces for Transfer Function Design with Semantics , 2006, IEEE Transactions on Visualization and Computer Graphics.

[83]  Carl-Fredrik Westin,et al.  Clustering Fiber Traces Using Normalized Cuts , 2004, MICCAI.

[84]  Markus Hadwiger,et al.  High-Quality Multimodal Volume Rendering for Preoperative Planning of Neurosurgical Interventions , 2007, IEEE Transactions on Visualization and Computer Graphics.

[85]  William M. Wells,et al.  Medical Image Computing and Computer-Assisted Intervention — MICCAI’98 , 1998, Lecture Notes in Computer Science.

[86]  Lukas Mroz,et al.  RTVR-a flexible Java library for interactive volume rendering , 2001, Proceedings Visualization, 2001. VIS '01..

[87]  B. M. ter Haar Romeny,et al.  Visualization of the fibrous structure of the heart , 2006 .

[88]  Ronald Fedkiw,et al.  Simulation of clothing with folds and wrinkles , 2003, SCA '03.

[89]  Markus Hadwiger,et al.  High-quality two-level volume rendering of segmented data sets on consumer graphics hardware , 2003, IEEE Visualization, 2003. VIS 2003..

[90]  Dinesh Manocha,et al.  OBBTree: a hierarchical structure for rapid interference detection , 1996, SIGGRAPH.

[91]  Anna Vilanova,et al.  Infants with perinatal hypoxic ischemia: feasibility of fiber tracking at birth and 3 months. , 2006, Radiology.

[92]  Nassir Navab,et al.  Respiratory motion analysis: Towards gated augmentation of the liver , 2005 .

[93]  Terry M. Peters,et al.  Mixed Reality Merging of Endoscopic Images and 3-D Surfaces , 2000, MICCAI.

[94]  Joe Michael Kniss,et al.  Statistically quantitative volume visualization , 2005, VIS 05. IEEE Visualization, 2005..

[95]  Christos Trantakis,et al.  Neuronavigation based on intraoperative 3D-ultrasound during tumor resection , 2005 .

[96]  Bernhard Preim,et al.  Visualization in Medicine: Theory, Algorithms, and Applications , 2007 .

[97]  Katja Bühler,et al.  Surface models of tube trees , 2004 .

[98]  Joe Michael Kniss,et al.  Gaussian transfer functions for multi-field volume visualization , 2003, IEEE Visualization, 2003. VIS 2003..

[99]  Dirk Bartz,et al.  Medical Augmented Reality based on Commercial Image Guided Surgery , 2004, EGVE.

[100]  Lukas Mroz,et al.  Advanced virtual endoscopic pituitary surgery , 2005, IEEE Transactions on Visualization and Computer Graphics.

[101]  Markus Hadwiger,et al.  Real-time volume graphics , 2006, Eurographics.

[102]  Carl-Fredrik Westin,et al.  Diffusion Tensor Visualization with Glyph Packing , 2006, IEEE Transactions on Visualization and Computer Graphics.

[103]  Anders Ynnerman,et al.  Uncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation , 2007, IEEE Transactions on Visualization and Computer Graphics.

[104]  Enrico Gobbetti,et al.  A single-pass GPU ray casting framework for interactive out-of-core rendering of massive volumetric datasets , 2008, The Visual Computer.

[105]  Gordon L. Kindlmann,et al.  Tensorlines: advection-diffusion based propagation through diffusion tensor fields , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[106]  Daniel Thalmann,et al.  An Adaptive Spatial Subdivision of the Object Space for Fast Collision Detection of Animated Rigid Bodies , 1995, Comput. Graph. Forum.

[107]  Anna Vilanova,et al.  Evaluation of fiber clustering methods for diffusion tensor imaging , 2005, VIS 05. IEEE Visualization, 2005..

[108]  Bernhard Preim,et al.  Real-Time Illustration of Vascular Structures , 2006, IEEE Transactions on Visualization and Computer Graphics.

[109]  Heinrich Niemann,et al.  Local 3D Reconstruction and Augmented Reality Visualization of Free-Hand Ultrasound for Needle Biopsy Procedures , 2003, MICCAI.

[110]  Heinrich Niemann,et al.  Single camera tracking of marker clusters: multiparameter cluster optimization and experimental verification , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[111]  P. Basser,et al.  Microstructural and physiological features of tissues elucidated by quantitative-diffusion-tensor MRI. 1996. , 1996, Journal of magnetic resonance.

[112]  J L Moctezuma,et al.  Intraoperative visualization of surgical planning data using video projectors. , 2001, Studies in health technology and informatics.

[113]  G. Roth,et al.  Hand tracking for interactive pattern-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[114]  Gabriel Zachmann,et al.  Collision Detection for Deformable Objects , 2004, Comput. Graph. Forum.

[115]  Vincent Lepetit,et al.  A semi-automatic method for resolving occlusion in augmented reality , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[116]  T. Ebbers,et al.  Particle trace visualization of intracardiac flow using time‐resolved 3D phase contrast MRI , 1999, Magnetic resonance in medicine.

[117]  Markus Hadwiger,et al.  Perspective Isosurface and Direct Volume Rendering for Virtual Endoscopy Applications , 2006, EuroVis.

[118]  Michael Gervautz,et al.  Occlusion in collaborative augmented environments , 1999, Comput. Graph..

[119]  Hanspeter Pfister,et al.  The VolumePro real-time ray-casting system , 1999, SIGGRAPH.

[120]  Amy A. Gooch,et al.  Graphics Interface 2007 , 2007 .

[121]  P. Basser,et al.  In vivo fiber tractography using DT‐MRI data , 2000, Magnetic resonance in medicine.

[122]  Bernhard Preim,et al.  International Journal of Computer Assisted Radiology and Surgery manuscript No. (will be inserted by the editor) Implicit Vessel Surface Reconstruction for Visualization and CFD Simulation , 2022 .

[123]  P. Sundgren,et al.  Diffusion tensor imaging of the brain: review of clinical applications , 2004, Neuroradiology.

[124]  Stefan Bruckner,et al.  Semantic Layers for Illustrative Volume Rendering , 2007, IEEE Transactions on Visualization and Computer Graphics.