A Virtual Reality System for Improved Image-Based Planning of Complex Cardiac Procedures

The intricate nature of congenital heart disease requires understanding of the complex, patient-specific three-dimensional dynamic anatomy of the heart, from imaging data such as three-dimensional echocardiography for successful outcomes from surgical and interventional procedures. Conventional clinical systems use flat screens, and therefore, display remains two-dimensional, which undermines the full understanding of the three-dimensional dynamic data. Additionally, the control of three-dimensional visualisation with two-dimensional tools is often difficult, so used only by imaging specialists. In this paper, we describe a virtual reality system for immersive surgery planning using dynamic three-dimensional echocardiography, which enables fast prototyping for visualisation such as volume rendering, multiplanar reformatting, flow visualisation and advanced interaction such as three-dimensional cropping, windowing, measurement, haptic feedback, automatic image orientation and multiuser interactions. The available features were evaluated by imaging and nonimaging clinicians, showing that the virtual reality system can help improve the understanding and communication of three-dimensional echocardiography imaging and potentially benefit congenital heart disease treatment.

[1]  David K. Harrison,et al.  Virtual Reality medical training system for anatomy education , 2014, 2014 Science and Information Conference.

[2]  T. Schaeffter,et al.  Towards a fast and efficient approach for modelling the patient-specific ventricular haemodynamics. , 2014, Progress in biophysics and molecular biology.

[3]  G. Biglino,et al.  Involving patients, families and medical staff in the evaluation of 3D printing models of congenital heart disease. , 2016, Communication & medicine.

[4]  I. Valverde,et al.  A systematic review of image segmentation methodology, used in the additive manufacture of patient-specific 3D printed models of the cardiovascular system , 2016, JRSM cardiovascular disease.

[5]  Jennifer N.A. Silva,et al.  Abstract 15358: Improving Visualization and Interaction During Transcatheter Ablation Using an Augmented Reality System: First-in-Human Experience , 2017 .

[6]  G. Marx,et al.  Three-dimensional echocardiography in congenital heart disease: an expert consensus document from the European Association of Cardiovascular Imaging and the American Society of Echocardiography. , 2016, European heart journal cardiovascular Imaging.

[7]  David Anderson,et al.  Three-dimensional printed models for surgical planning of complex congenital heart defects: an international multicentre study , 2017, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[8]  Srinivasan Krishnaswamy,et al.  A Semi-automated Approach to Improve the Efficiency of Medical Imaging Segmentation for Haptic Rendering , 2017, Journal of Digital Imaging.

[9]  Dieter Schmalstieg,et al.  HTC Vive MeVisLab integration via OpenVR for medical applications , 2017, PloS one.

[10]  Narutoshi Hibino,et al.  Role of virtual reality in congenital heart disease , 2018, Congenital heart disease.

[11]  Reza Nezafat,et al.  Three-dimensional holographic visualization of high-resolution myocardial scar on HoloLens , 2018, PloS one.

[12]  Nicolas Toussaint,et al.  Virtual interaction and visualisation of 3D medical imaging data with VTK and Unity , 2018, Healthcare technology letters.

[13]  Yiyu Cai,et al.  Augmented Reality Simulation of Cardiac Circulation Using APPLearn (Heart) , 2018, 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

[14]  Yihan Lin,et al.  Using Touch Surgery to Improve Surgical Education in Low- and Middle-Income Settings: A Randomized Control Trial. , 2017, Journal of surgical education.

[15]  Mark Asselin,et al.  Interaction with Volume-Rendered Three-Dimensional Echocardiographic Images in Virtual Reality. , 2018, Journal of the American Society of Echocardiography : official publication of the American Society of Echocardiography.

[16]  S. Birkeland,et al.  Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease. , 2018, European heart journal cardiovascular Imaging.

[17]  Jennifer N.A. Silva,et al.  Emerging Applications of Virtual Reality in Cardiovascular Medicine , 2018, JACC. Basic to translational science.

[18]  Julia A. Schnabel,et al.  Unity and VTK for VR Medical Image Analysis-an Initial Clinical Evaluation , 2019 .

[19]  M. Pakkal,et al.  Virtual reality and cardiac anatomy: Exploring immersive three‐dimensional cardiac imaging, a pilot study in undergraduate medical anatomy education , 2018, Clinical anatomy.

[20]  Julia A. Schnabel,et al.  Virtual linear measurement system for accurate quantification of medical images , 2019, Healthcare technology letters.

[21]  M. Meineri,et al.  Validation of Quantitative 3-Dimensional Transesophageal Echocardiography Mitral Valve Analysis Using Stereoscopic Display. , 2019, Journal of cardiothoracic and vascular anesthesia.

[22]  Tania Bardyn,et al.  Advancing Cardiac Surgery Case Planning and Case Review Conferences Using Virtual Reality in Medical Libraries: Evaluation of the Usability of Two Virtual Reality Apps , 2019, JMIR human factors.

[23]  Mark Asselin,et al.  SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality , 2020, IEEE Transactions on Medical Robotics and Bionics.

[24]  D. Rueckert,et al.  Deep Learning for Cardiac Image Segmentation: A Review , 2019, Frontiers in Cardiovascular Medicine.

[25]  J. Schnabel,et al.  P1417 Acceptability of a virtual reality system for examination of congenital heart disease patients , 2020 .

[26]  J. Schnabel,et al.  P801 A virtual reality tool for measurement of 3D echocardiographic images , 2020 .

[27]  J. Simpson,et al.  3D hybrid printed models in complex congenital heart disease: 3D echocardiography and cardiovascular magnetic resonance imaging fusion. , 2020, European heart journal.

[28]  Brant G. Miller,et al.  Augmented reality-based learning for the comprehension of cardiac physiology in undergraduate biomedical students. , 2020, Advances in physiology education.

[29]  J. Schnabel,et al.  P1566 Evaluation of haptic feedback for interaction with volumetric image data in virtual reality , 2020 .

[30]  J. Schnabel,et al.  Virtual reality three-dimensional echocardiographic imaging for planning surgical atrioventricular valve repair , 2021, JTCVS techniques.

[31]  Julia A. Schnabel,et al.  Automatic Re-orientation of 3D Echocardiographic Images in Virtual Reality Using Deep Learning , 2021, MIUA.

[32]  J. Schnabel,et al.  Automatic orientation cues for intuitive immersive interrogation of 3D echocardiographic images in virtual reality using deep learning , 2021 .

[33]  J. Schnabel,et al.  Immersive visualisation of intracardiac blood flow in virtual reality on a patient with HLHS , 2021 .