Applying Modern Virtual and Augmented Reality Technologies to Medical Images and Models

Recent technological innovations have created new opportunities for the increased adoption of virtual reality (VR) and augmented reality (AR) applications in medicine. While medical applications of VR have historically seen greater adoption from patient-as-user applications, the new era of VR/AR technology has created the conditions for wider adoption of clinician-as-user applications. Historically, adoption to clinical use has been limited in part by the ability of the technology to achieve a sufficient quality of experience. This article reviews the definitions of virtual and augmented reality and briefly covers the history of their development. Currently available options for consumer-level virtual and augmented reality systems are presented, along with a discussion of technical considerations for their adoption in the clinical environment. Finally, a brief review of the literature of medical VR/AR applications is presented prior to introducing a comprehensive conceptual framework for the viewing and manipulation of medical images in virtual and augmented reality. Using this framework, we outline considerations for placing these methods directly into a radiology-based workflow and show how it can be applied to a variety of clinical scenarios.

[1]  Christian Hansen,et al.  Highly immersive virtual reality laparoscopy simulation: development and future aspects , 2018, International Journal of Computer Assisted Radiology and Surgery.

[2]  Stephan Arens,et al.  A Survey of Transfer Functions Suitable for Volume Rendering , 2010, VG@Eurographics.

[3]  Todd Pietila,et al.  Creating patient-specific anatomical models for 3D printing and AR/VR: a supplement for the 2018 Radiological Society of North America (RSNA) hands-on course , 2019, 3D Printing in Medicine.

[4]  R. Satava.  Virtual reality surgical simulator , 1993, Surgical Endoscopy.

[5]  G. Fichtinger,et al.  An Immersive Virtual Reality Environment for Diagnostic Imaging , 2016, J. Medical Robotics Res..

[6]  Brenda K. Wiederhold,et al.  Virtual reality therapy for anxiety disorders: Advances in evaluation and treatment. , 2005 .

[7]  Marco Ferrari,et al.  Recent Trends and Future Direction of Dental Research in the Digital Era , 2020, International journal of environmental research and public health.

[8]  Woodrow Barfield.  Visual Consistency in Augmented Reality Compositing , 2015 .

[9]  Donald R. Laub,et al.  Evolution of virtual reality [Medicine] , 1996 .

[10]  Linping Zhao,et al.  Accuracy and Reproducibility of Linear and Angular Measurements in Virtual Reality: a Validation Study , 2019, Journal of Digital Imaging.

[11]  F. Rybicki,et al.  Medical 3D Printing for the Radiologist. , 2015, Radiographics : a review publication of the Radiological Society of North America, Inc.

[12]  Frank J Rybicki,et al.  Message From Frank J. Rybicki, MD, Incoming Chair of ACR Appropriateness Criteria. , 2017, Journal of the American College of Radiology : JACR.

[13]  F. Rybicki,et al.  Applications of 3D printing in cardiovascular diseases , 2016, Nature Reviews Cardiology.

[14]  D. Louis Collins,et al.  Augmented reality in neurovascular surgery: feasibility and first uses in the operating room , 2015, International Journal of Computer Assisted Radiology and Surgery.

[15]  Hunter G Hoffman,et al.  The Effect of Virtual Reality on Pain and Range of Motion in Adults With Burn Injuries , 2009, Journal of burn care & research : official publication of the American Burn Association.

[16]  Jens H. Krüger,et al.  State of the Art in Transfer Functions for Direct Volume Rendering , 2016, Comput. Graph. Forum.

[17]  Q. Louw,et al.  Feasibility and potential effect of a low-cost virtual reality system on reducing pain and anxiety in adult burn injury patients during physiotherapy in a developing country. , 2010, Burns : journal of the International Society for Burn Injuries.

[18]  Rüdiger Westermann,et al.  Acceleration techniques for GPU-based volume rendering , 2003, IEEE Visualization, 2003. VIS 2003..

[19]  Dimitrios Mitsouras,et al.  D Printing Technologies , 2019 .

[20]  David B. Douglas,et al.  Virtual reality and augmented reality: Advances in surgery , 2017 .

[21]  Andy Christensen,et al.  Maintaining safety and efficacy for 3D printing in medicine , 2017, 3D Printing in Medicine.

[22]  G. Riva,et al.  Virtual Reality for Enhancing the Cognitive Behavioral Treatment of Obesity With Binge Eating Disorder: Randomized Controlled Study With One-Year Follow-up , 2013, Journal of medical Internet research.

[23]  Heikki Lehväslaiho,et al.  Three‐dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues , 2015, The Journal of comparative neurology.

[24]  Raúl San José Estépar,et al.  Real‐time computed tomography‐based augmented reality for natural orifice transluminal endoscopic surgery navigation , 2012, The British journal of surgery.

[25]  Morton L. Heilig,et al.  United States Patent Office: stereoscopic-television apparatus for individual use , 1994, COMG.

[26]  Matthew Di Prima,et al.  Additively manufactured medical products – the FDA perspective , 2016, 3D Printing in Medicine.

[27]  Simon Stegmaier,et al.  A simple and flexible volume rendering framework for graphics-hardware-based raycasting , 2005, Fourth International Workshop on Volume Graphics, 2005..

[28]  James L Patton,et al.  Tolerance of a virtual reality intervention for attention remediation in persons with severe TBI , 2011, Brain injury.

[29]  Howard Rheingold,et al.  Virtual Reality , 1991 .

[30]  E. Ho,et al.  The effectiveness of therapeutic play, using virtual reality computer games, in promoting the psychological well-being of children hospitalised with cancer. , 2011, Journal of clinical nursing.

[31]  Andrew R. Lilja,et al.  Journey to the centre of the cell: Virtual reality immersion into scientific data , 2018, Traffic.

[32]  Nassir Navab,et al.  On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic , 2013, Comput. Medical Imaging Graph..

[33]  Liron Pantanowitz,et al.  Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology. , 2018, Archives of pathology & laboratory medicine.

[34]  Charles B. Owen,et al.  Review on cybersickness in applications and visual displays , 2016, Virtual Reality.

[35]  Bradford J. Wood,et al.  Augmented and Mixed Reality: Technologies for Enhancing the Future of IR. , 2020, Journal of vascular and interventional radiology : JVIR.

[36]  Wei Zhang,et al.  A Unified Framework for Street-View Panorama Stitching , 2016, Sensors.

[37]  Frank J. Rybicki.  3D Printing in Medicine: an introductory message from the Editor-in-Chief , 2015, 3D Printing in Medicine.

[38]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[39]  Sam R. Sharar,et al.  Virtual Reality Hypnosis for Pain Associated With Recovery From Physical Trauma , 2010, The International journal of clinical and experimental hypnosis.

[40]  Richard M. Satava,et al.  Current and future applications of virtual reality for medicine , 1998, Proc. IEEE.

[41]  Hunter G Hoffman,et al.  A randomized, controlled trial of immersive virtual reality analgesia, during physical therapy for pediatric burns. , 2011, Burns : journal of the International Society for Burn Injuries.

[42]  A. Z. Syed,et al.  DARK ROOM TO AUGMENTED REALITY: APPLICATION OF HOLOLENS TECHNOLOGY FOR ORAL RADIOLOGICAL DIAGNOSIS , 2017 .

[43]  Robert Teasell,et al.  Efficacy and safety of non-immersive virtual reality exercising in stroke rehabilitation (EVREST): a randomised, multicentre, single-blind, controlled trial , 2016, The Lancet Neurology.

[44]  M. Slater,et al.  Virtual reality in the assessment, understanding, and treatment of mental health disorders , 2017, Psychological Medicine.

[45]  Daniel Tazartes,et al.  An historical perspective on inertial navigation systems , 2014, 2014 International Symposium on Inertial Sensors and Systems (ISISS).

[46]  K. Weimer,et al.  Mixed Reality with HoloLens: Where Virtual Reality Meets Augmented Reality in the Operating Room , 2017, Plastic and reconstructive surgery.

[47]  Fengfeng Zhang,et al.  Optimization of virtual and real registration technology based on augmented reality in a surgical navigation system , 2020, BioMedical Engineering OnLine.

[48]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[49]  Heikki Lehväslaiho,et al.  Three-dimensional immersive virtual reality for studying cellular compartments in 3D models from EM preparations of neural tissues. , 2016, The Journal of comparative neurology.

[50]  Ziv Yaniv,et al.  Applications of Augmented Reality in the Operating Room , 2015 .

[51]  K Gallagher,et al.  Making and viewing stereoscopic surgical videos with smartphones and virtual reality headset , 2016, Eye.

[52]  Andrea Gaggioli,et al.  Being different: The transformative potential of virtual reality , 2016 .

[53]  L. Roberts,et al.  Comparing In-Person to Videoconference-Based Cognitive Behavioral Therapy for Mood and Anxiety Disorders: Randomized Controlled Trial , 2013, Journal of medical Internet research.

[54]  Diego Martínez Plasencia.  One step beyond virtual reality , 2015, XRDS.

[55]  Marc Levoy,et al.  Display of surfaces from volume data , 1988, IEEE Computer Graphics and Applications.

[56]  Ramon Farré,et al.  Alzheimer’s Disease Mutant Mice Exhibit Reduced Brain Tissue Stiffness Compared to Wild-type Mice in both Normoxia and following Intermittent Hypoxia Mimicking Sleep Apnea , 2018, Front. Neurol..

[57]  Frank J. Rybicki,et al.  3D Printing in Medicine , 2017, Springer International Publishing.

[58]  Anahi Perlas,et al.  A Virtual Reality Simulation Model of Spinal Ultrasound: Role in Teaching Spinal Sonoanatomy , 2016, Regional Anesthesia & Pain Medicine.

[59]  Frank J Rybicki.  Medical 3D printing and the physician-artist , 2018, The Lancet.

[60]  I. Danovitch,et al.  Virtual Reality and Medical Inpatients: A Systematic Review of Randomized, Controlled Trials. , 2017, Innovations in clinical neuroscience.

[61]  Hunter G. Hoffman,et al.  The Use of Virtual Reality to Facilitate Mindfulness Skills Training in Dialectical Behavioral Therapy for Borderline Personality Disorder: A Case Study , 2016, Front. Psychol..

[62]  J. Boehmler.  Virtual reality for acute pain reduction in adolescents undergoing burn wound care: A prospective randomized controlled trial , 2013 .

[63]  G. Andersson,et al.  Single-session gamified virtual reality exposure therapy for spider phobia vs. traditional exposure therapy: study protocol for a randomized controlled non-inferiority trial , 2016, Trials.

[64]  Claudio Pensieri,et al.  Overview: Virtual Reality in Medicine , 2014 .

[65]  Barbara O Rothbaum,et al.  Investigating Relationships Between PTSD Symptom Clusters Within Virtual Reality Exposure Therapy for OEF/OIF Veterans. , 2017, Behavior therapy.

[66]  Oscar Meruvia Pastor,et al.  Augmented Reality as a Telemedicine Platform for Remote Procedural Training , 2017, Sensors.

[67]  Carl Machover,et al.  Virtual reality , 1994, IEEE Computer Graphics and Applications.

[68]  Dimitrios Mitsouras,et al.  3D Printing Technologies , 2022 .

[69]  Carina W. Yang,et al.  Virtual Simulation in Enhancing Procedural Training for Fluoroscopy-guided Lumbar Puncture: A Pilot Study. , 2017, Academic radiology.

[70]  Luo Luo Zheng,et al.  Mobile Virtual Reality for Ophthalmic Image Display and Diagnosis , 2015 .

[71]  A. Cuschieri,et al.  A Systematic Review of Virtual Reality Simulators for Robot-assisted Surgery. , 2016, European urology.

[72]  C Chinnock,et al.  Virtual reality in surgery and medicine. , 1994, Hospital technology series.

[73]  Rajesh Aggarwal,et al.  Psychomotor skills and cognitive load training on a virtual reality laparoscopic simulator for tubal surgery is effective. , 2013, European journal of obstetrics, gynecology, and reproductive biology.

[74]  Giuseppe Riva,et al.  Can relaxation training reduce emotional eating in women with obesity? An exploratory study with 3 months of follow-up. , 2009, Journal of the American Dietetic Association.

[75]  Simon Weidert,et al.  Intra-operative augmented reality in distal locking , 2015, International Journal of Computer Assisted Radiology and Surgery.

[76]  Lisa Rebenitsch,et al.  Managing cybersickness in virtual reality , 2015, XRDS.

[77]  Karla Hemming,et al.  Proceedings of the First International Conference on Stepped Wedge Trial Design , 2016, Trials.

[78]  J R Phillips.  Virtual Reality: A New Vista for Nurse Researchers? , 1993, Nursing science quarterly.

[79]  C. Moro,et al.  The effectiveness of virtual and augmented reality in health sciences and medical anatomy , 2017, Anatomical sciences education.

[80]  Philippe Coiffet,et al.  Virtual Reality Technology , 2003, Presence: Teleoperators & Virtual Environments.

[81]  Elizabeth A. Krupinski,et al.  ACR–AAPM–SIIM Technical Standard for Electronic Practice of Medical Imaging , 2013, Journal of Digital Imaging.

[82]  H. Hoffman,et al.  Virtual Reality Pain Control During Burn Wound Debridement in the Hydrotank , 2008, The Clinical journal of pain.

[83]  Katherine J. Kuchenbecker,et al.  Immersive Low-Cost Virtual Reality Treatment for Phantom Limb Pain: Evidence from Two Cases , 2018, Front. Neurol..

[84]  David B Douglas,et al.  D3D augmented reality imaging system: proof of concept in mammography , 2016, Medical devices.

[85]  Dimitrios Mitsouras,et al.  Medical 3D printing: methods to standardize terminology and report trends , 2017, 3D Printing in Medicine.

[86]  Stefano Tamburin,et al.  Psychological Considerations in the Assessment and Treatment of Pain in Neurorehabilitation and Psychological Factors Predictive of Therapeutic Response: Evidence and Recommendations from the Italian Consensus Conference on Pain in Neurorehabilitation , 2016, Front. Psychol..

[87]  Brian Silver.  Virtual reality versus reality in post-stroke rehabilitation , 2016, The Lancet Neurology.

[88]  Michael Harrington,et al.  Constellation: a wide-range wireless motion-tracking system for augmented reality and virtual set applications , 1998, SIGGRAPH.

[89]  Anthony G. Gallagher,et al.  Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators , 2016, BMJ Simulation & Technology Enhanced Learning.