Medical Augmented-Reality Visualizer for Surgical Training and Education in Medicine

This paper presents a projection-based augmented-reality system (MARVIS) that supports the visualization of internal structures on the surface of a liver phantom. MARVIS is endowed with three key features: tracking of spatial relationship between the phantom and the operator’s head in real time, monoscopic projection of internal liver structures onto the phantom surface for 3D perception without additional head-mounted devices, and phantom internal electronic circuit to assess the accuracy of a syringe guidance system. An initial validation was carried out by 25 medical students (12 males and 13 females; mean age, 23.12 years; SD, 1.27 years) and 3 male surgeons (mean age, 43.66 years; SD, 7.57 years). The validation results show that the ratio of failed syringe insertions was reduced from 50% to 30% by adopting the MARVIS projection. The proposed system suitably enhances a surgeon’s spatial perception of a phantom internal structure.

[1]  Wolfgang Straßer,et al.  3D Surveillance A Distributed Network of Smart Cameras for Real-Time Tracking and its Visualization in 3D , 2006, 2006 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'06).

[2]  Gerd Bruder,et al.  Physical-Virtual Agents for Healthcare Simulation , 2018, IVA.

[3]  Jorge Martín-Gutiérrez,et al.  Augmented reality to promote collaborative and autonomous learning in higher education , 2015, Comput. Hum. Behav..

[4]  Niels De Ruiter,et al.  A Hybrid 2D/3D User Interface for Radiological Diagnosis , 2018, Journal of Digital Imaging.

[5]  M. Schijven,et al.  The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review , 2009, Surgical Endoscopy.

[6]  Gregory Kramida,et al.  Resolving the Vergence-Accommodation Conflict in Head-Mounted Displays , 2016, IEEE Transactions on Visualization and Computer Graphics.

[7]  John C. Tang,et al.  3D Collaboration Method over HoloLens™ and Skype™ End Points , 2015, ImmersiveME@ACM Multimedia.

[8]  Kangdon Lee,et al.  Augmented Reality in Education and Training , 2012, TechTrends.

[9]  Thierry Duval,et al.  Dealing with frame cancellation for stereoscopic displays in 3D user interfaces , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[10]  L. Jain,et al.  Virtual, Augmented Reality and Serious Games for Healthcare 1 , 2014 .

[11]  Heinrich H. Bülthoff,et al.  Perceiving simulated ego-motions in virtual reality: comparing large screen displays with HMDs , 2005, IS&T/SPIE Electronic Imaging.

[12]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  C. Blakemore.  The range and scope of binocular depth discrimination in man , 1970, The Journal of physiology.

[14]  Derek R. Magee,et al.  An augmented reality simulator for ultrasound guided needle placement training , 2007, Medical & Biological Engineering & Computing.

[15]  Narendra Ahuja,et al.  Using a head-mounted projective display in interactive augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[16]  David Fonseca,et al.  Relationship between student profile, tool use, participation, and academic performance with the use of Augmented Reality technology for visualized architecture models , 2014, Comput. Hum. Behav..

[17]  David M. Hoffman,et al.  Vergence-accommodation conflicts hinder visual performance and cause visual fatigue. , 2008, Journal of vision.

[18]  F. Okano,et al.  Repeated vergence adaptation causes the decline of visual functions in watching stereoscopic television , 2005, Journal of Display Technology.

[19]  Luca Vigano,et al.  The learning curve in laparoscopic major liver resection , 2013, Journal of hepato-biliary-pancreatic sciences.

[20]  P. Dasgupta,et al.  Simulation training in upper tract endourology: myth or reality? , 2017, Minerva urologica e nefrologica = The Italian journal of urology and nephrology.

[21]  Robert Sitnik,et al.  Robust geometric, phase and colour structured light projection system calibration , 2017 .

[22]  Jorge Bacca,et al.  Augmented Reality Trends in Education: A Systematic Review of Research and Applications , 2014, J. Educ. Technol. Soc..

[23]  K. J. Kuchenbecker,et al.  Surgeons and non-surgeons prefer haptic feedback of instrument vibrations during robotic surgery , 2015, Surgical Endoscopy.

[24]  R. Lloréns,et al.  Comparison of Oculus Rift and HTC Vive: Feasibility for Virtual Reality-Based Exploration, Navigation, Exergaming, and Rehabilitation. , 2018, Games for health journal.

[25]  W. N. Charman,et al.  Pupil Diameter and the Depth-of-field of the Human Eye as Measured by Laser Speckle , 1977 .

[26]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Omar M. Jeroudi,et al.  Accuracy of remote electrocardiogram interpretation with the use of Google Glass technology. , 2015, The American journal of cardiology.

[28]  Liron Pantanowitz,et al.  Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology. , 2018, Archives of pathology & laboratory medicine.

[29]  Axel Pinz,et al.  A new optical tracking system for virtual and augmented reality applications , 2001, IMTC 2001. Proceedings of the 18th IEEE Instrumentation and Measurement Technology Conference. Rediscovering Measurement in the Age of Informatics (Cat. No.01CH 37188).

[30]  Yasuhiro Takaki.  Novel 3D display using an array of LCD panels , 2003, IS&T/SPIE Electronic Imaging.

[31]  B. Krug,et al.  Application of an augmented reality tool for maxillary positioning in orthognathic surgery - a feasibility study. , 2006, Journal of cranio-maxillo-facial surgery : official publication of the European Association for Cranio-Maxillo-Facial Surgery.

[32]  Walterio W. Mayol-Cuevas,et al.  A first approach to tactile texture recognition , 1998, SMC'98 Conference Proceedings. 1998 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.98CH36218).

[33]  Panteleimon Pantelidis,et al.  Virtual and Augmented Reality in Medical Education , 2018 .

[34]  Darwin G. Caldwell,et al.  Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation , 2011, IEEE Transactions on Haptics.

[35]  W. Lewis,et al.  Learning curves in surgical practice , 2007, Postgraduate Medical Journal.

[36]  B. Julesz.  Foundations of Cyclopean Perception , 1971 .

[37]  C. Schor,et al.  Binocular sensory fusion is limited by spatial resolution , 1984, Vision Research.

[38]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).