Applications of Augmented Reality in the Operating Room

Most surgical procedures and therapeutic interventions have been traditionally performed by gaining direct access to the internal anatomy, using direct visual inspection to deliver therapy and treat the conditions. In the meantime, a wide variety of medical imaging modalities have been employed to diagnose the condition, plan the procedure and monitor the patient during the intervention; nevertheless, the tissue manipulation and delivery of therapy has been performed under rather invasive incisions that permit direct visualization of the surgical site and ample access inside the body cavity. Over the past several decades, significant efforts have been dedicated to minimizing invasiveness associated with surgical interventions, most of which have been possible thanks to the developments in medical imaging, surgical navigation, visualization and display technologies.

[1]  Terry M. Peters,et al.  Evaluation of Model-Enhanced Ultrasound-Assisted Interventional Guidance in a Cardiac Phantom , 2010, IEEE Transactions on Biomedical Engineering.

[2]  Russell H. Taylor,et al.  Cadaveric feasibility study of da Vinci Si-assisted cochlear implant with augmented visual navigation for otologic surgery. , 2014, JAMA Otolaryngology - Head and Neck Surgery.

[3]  Frank Sauer,et al.  An augmented reality system for MR image-guided needle biopsy: initial results in a swine model. , 2006, Radiology.

[4]  D. Koulalis,et al.  Computer-assisted navigation in orthopedic surgery. , 2013, Orthopedics.

[5]  Ren Hui Gong,et al.  Interactive initialization of 2D/3D rigid registration. , 2013, Medical physics.

[6]  H. Meinzer,et al.  Augmented reality visualization during laparoscopic radical prostatectomy. , 2011, Journal of endourology.

[7]  Paul Suetens,et al.  An augmented reality system for patient-specific guidance of cardiac catheter ablation procedures , 2005, IEEE Transactions on Medical Imaging.

[8]  Thomas P. Caudell Introduction to augmented and virtual reality , 1995, Other Conferences.

[9]  H. Feußner,et al.  [Laparoscopic ultrasound]. , 2007, Der Chirurg; Zeitschrift fur alle Gebiete der operativen Medizen.

[10]  Lejing Wang,et al.  Precise X-ray and video overlay for augmented reality fluoroscopy , 2012, International Journal of Computer Assisted Radiology and Surgery.

[11]  Sebastian Bodenstedt,et al.  A system for context-aware intraoperative augmented reality in dental implant surgery , 2014, International Journal of Computer Assisted Radiology and Surgery.

[12]  Louis B. Rosenberg,et al.  Virtual fixtures: Perceptual tools for telerobotic manipulation , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[13]  D. Simon,et al.  Virtual Fluoroscopy: Computer-Assisted Fluoroscopic Navigation , 2001, Spine.

[14]  Thomas Bodenheimer,et al.  High and Rising Health Care Costs. Part 2: Technologic Innovation , 2005, Annals of Internal Medicine.

[15]  M. Conditt,et al.  Robotic Arm-assisted UKA Improves Tibial Component Alignment: A Pilot Study , 2010, Clinical orthopaedics and related research.

[16]  Neil D. Glossop,et al.  Laser projection augmented reality system for computer-assisted surgery , 2003, CARS.

[17]  George D. Stetten,et al.  Intuitive Intraoperative Ultrasound Guidance Using the Sonic Flashlight: A Novel Ultrasound Display System , 2005, Neurosurgery.

[18]  Paul J. Metzger Adding reality to the virtual , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[19]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[20]  Örjan Smedby,et al.  Real-time intraoperative visualization of myocardial circulation using augmented reality temperature display , 2013, The International Journal of Cardiovascular Imaging.

[21]  S D Adhikary,et al.  Simulator for teaching hand-eye coordination during ultrasound-guided regional anaesthesia. , 2013, British journal of anaesthesia.

[22]  David J. Hawkes,et al.  Design and evaluation of a system for microscope-assisted guided interventions (MAGI) , 2000 .

[23]  Michael J. Daly,et al.  Surgeons blinded by enhanced navigation: the effect of augmented reality on attention , 2013, Surgical Endoscopy.

[24]  J P Rolland,et al.  Towards a novel augmented-reality tool to visualize dynamic 3-D anatomy. , 1997, Studies in health technology and informatics.

[25]  O. Ratib,et al.  Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion , 2011, Journal of hepato-biliary-pancreatic sciences.

[26]  M Shoham,et al.  Image-guided system with miniature robot for precise positioning and targeting in keyhole neurosurgery , 2006, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[27]  M. Fogel,et al.  X-Ray Magnetic Resonance Fusion to Internal Markers and Utility in Congenital Heart Disease Catheterization , 2011, Circulation. Cardiovascular imaging.

[28]  Michael J. Daly,et al.  Augmented Real-Time Navigation with Critical Structure Proximity Alerts for Endoscopic Skull Base Surgery , 2012 .

[29]  Luc Soler,et al.  An augmented reality system for liver thermal ablation: Design and evaluation on clinical cases , 2009, Medical Image Anal..

[30]  R. Tamburo,et al.  Guidance of Retrobulbar Injection With Real‐time Tomographic Reflection , 2002, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[31]  K. Cleary,et al.  Image-guided interventions: technology review and clinical applications. , 2010, Annual review of biomedical engineering.

[32]  Heinrich Niemann,et al.  An AR system with intuitive user interface for manipulation and visualization of 3D medical data. , 2004, Studies in health technology and informatics.

[33]  Jenny Dankelman,et al.  Discovery of high-level tasks in the operating room , 2011, J. Biomed. Informatics.

[34]  S. Nitzsche,et al.  Cardio navigation: planning, simulation, and augmented reality in robotic assisted endoscopic bypass grafting. , 2005, The Annals of thoracic surgery.

[35]  R. Kikinis,et al.  A new visualization technique for laparoscopic ultrasonography. , 2004, Surgery.

[36]  H Wörn,et al.  New augmented reality and robotic based methods for head‐surgery , 2005, The international journal of medical robotics + computer assisted surgery : MRCAS.

[37]  Roberta Klatzky,et al.  Peripherally inserted central catheter placement using the Sonic Flashlight. , 2009, Journal of vascular and interventional radiology : JVIR.

[38]  Masahide Kaneko,et al.  Toward the New Era of Visual Communication (Special Issue on Next Generation Visual Telecommunication and Broadcasting) , 1993 .

[39]  C Baur,et al.  A non-contact mouse for surgeon-computer interaction. , 2004, Technology and health care : official journal of the European Society for Engineering and Medicine.

[40]  Frank Sauer,et al.  Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation , 2006, International Journal of Computer Vision.

[41]  Nassir Navab,et al.  Statistical modeling and recognition of surgical workflow , 2012, Medical Image Anal..

[42]  Tim Lüth,et al.  The Navigated Image Viewer - Evaluation in Maxillofacial Surgery , 2003, MICCAI.

[43]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[44]  G. Stetten,et al.  Overlaying ultrasonographic images on direct vision. , 2001, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[45]  Terry M. Peters,et al.  A Navigation Platform for Guidance of Beating Heart Transapical Mitral Valve Repair , 2013, IEEE Transactions on Biomedical Engineering.

[46]  Maurice Anidjar,et al.  Hybrid augmented reality simulator: preliminary construct validation of laparoscopic smoothness in a urology residency program. , 2008, The Journal of urology.

[47]  B. Krug,et al.  Application of an augmented reality tool for maxillary positioning in orthognathic surgery - a feasibility study. , 2006, Journal of cranio-maxillo-facial surgery : official publication of the European Association for Cranio-Maxillo-Facial Surgery.

[48]  M Jakopec,et al.  Robotic clinical trials of uni‐condylar arthroplasty , 2005, The international journal of medical robotics + computer assisted surgery : MRCAS.

[49]  Ken Masamune,et al.  Image overlay guidance for needle insertion in CT scanner , 2005, IEEE Transactions on Biomedical Engineering.

[50]  Nassir Navab,et al.  Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures , 2007, IEEE Computer Graphics and Applications.

[51]  Eduardo E. Lovo,et al.  A NOVEL, INEXPENSIVE METHOD OF IMAGE COREGISTRATION FOR APPLICATIONS IN IMAGE‐GUIDED SURGERY USING AUGMENTED REALITY , 2007, Neurosurgery.

[52]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[53]  Michael V Knopp,et al.  A comprehensive overview of radioguided surgery using gamma detection probe technology , 2009, World journal of surgical oncology.

[54]  Stefan Weber,et al.  A Portable Image Overlay Projection Device for Computer-Aided Open Liver Surgery , 2011, IEEE Transactions on Biomedical Engineering.

[55]  Mohammad-Reza Siadat,et al.  Design, implementation and accuracy of a prototype for medical augmented reality , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[56]  Simon P. DiMaio,et al.  The da Vinci Surgical System , 2011 .

[57]  W. Eric L. Grimson,et al.  Experimentation with a transcranial magnetic stimulation system for functional brain mapping , 1998, Medical Image Anal..

[58]  T. Vogl,et al.  Navigation-Based Needle Puncture of a Cadaver Using a Hybrid Tracking Navigational System , 2006, Investigative radiology.

[59]  Dieter Schmalstieg,et al.  Comprehensible Visualization for Augmented Reality , 2009, IEEE Transactions on Visualization and Computer Graphics.

[60]  Stefan Weber,et al.  Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies , 2012, International Journal of Computer Assisted Radiology and Surgery.

[61]  Purang Abolmaesumi,et al.  An Augmented Reality System for Epidural Anesthesia (AREA): Prepuncture Identification of Vertebrae , 2013, IEEE Transactions on Biomedical Engineering.

[62]  P. Rowe,et al.  Time analysis of MAKO RIO UKA procedures in comparision with the Oxford UKA , 2013 .

[63]  S. Kreuer,et al.  The Performance of Six Pulse Oximeters in the Environment of Neuronavigation , 2008, Anesthesia and analgesia.

[64]  Anne Auperin,et al.  Electromagnetic-Tracked Biopsy under Ultrasound Guidance: Preliminary Results , 2012, CardioVascular and Interventional Radiology.

[65]  Pierre Jannin,et al.  Surgical models for computer-assisted neurosurgery , 2007, NeuroImage.

[66]  Lena Maier-Hein,et al.  Mobile augmented reality for computer-assisted percutaneous nephrolithotomy , 2013, International Journal of Computer Assisted Radiology and Surgery.

[67]  Makoto Hashizume,et al.  Intraoperative Magnetic Tracker Calibration Using a Magneto-Optic Hybrid Tracker for 3-D Ultrasound-Based Navigation in Laparoscopic Surgery , 2008, IEEE Transactions on Medical Imaging.

[68]  Jean-Luc Bosson,et al.  Computer-assisted navigation in ACL reconstruction is attractive but not yet cost efficient , 2015, Knee Surgery, Sports Traumatology, Arthroscopy.

[69]  W. Thumfart,et al.  Image-guided endoscopic ENT surgery , 2007, European Archives of Oto-Rhino-Laryngology.

[70]  Emad M. Boctor,et al.  Navigation with local sensors in handheld 3D ultrasound: initial in-vivo experience , 2011, Medical Imaging.

[71]  Kevin Cleary,et al.  Stereoscopic augmented reality for laparoscopic surgery , 2014, Surgical Endoscopy.

[72]  Nassir Navab,et al.  First demonstration of 3-D lymphatic mapping in breast cancer using freehand SPECT , 2010, European Journal of Nuclear Medicine and Molecular Imaging.

[73]  Tomas Lozano-Perez,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1996 .

[74]  Yoshikazu Nakajima,et al.  Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization , 1998, IEEE Transactions on Medical Imaging.

[75]  Edit Varga,et al.  Manipulation of mental models of anatomy in interventional radiology and its consequences for design of human–computer interaction , 2013, Cognition, Technology & Work.

[76]  A. Park,et al.  Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography , 2010, Surgical Endoscopy.

[77]  Heinz-Otto Peitgen,et al.  Illustrative visualization of 3D planning models for augmented reality in liver surgery , 2010, International Journal of Computer Assisted Radiology and Surgery.

[78]  Nassir Navab,et al.  Camera Augmented Mobile C-Arm (CAMC): Calibration, Accuracy Study, and Clinical Applications , 2010, IEEE Transactions on Medical Imaging.

[79]  John B Martinie,et al.  Novel 3-D laparoscopic magnetic ultrasound image guidance for lesion targeting. , 2010, HPB : the official journal of the International Hepato Pancreato Biliary Association.

[80]  Michael Conditt,et al.  Technology and cost-effectiveness in knee arthroplasty: computer navigation and robotics. , 2009, American journal of orthopedics.

[81]  Maurizio Fornari,et al.  Economic study: a cost-effectiveness analysis of an intraoperative compared with a preoperative image-guided system in lumbar pedicle screw fixation in patients with degenerative spondylolisthesis. , 2014, The spine journal : official journal of the North American Spine Society.

[82]  Technical aspects and feasibility of laparoscopic ultrasound navigation in radiofrequency ablation of unresectable hepatic malignancies. , 2007, Journal of laparoendoscopic & advanced surgical techniques. Part A.

[83]  Harald Hoppe,et al.  New Augmented Reality Concepts for Craniofacial Surgical Procedures , 2005, Plastic and reconstructive surgery.

[84]  A. J. Floyd,et al.  Robotic systems in orthopaedic surgery. , 2011, The Journal of bone and joint surgery. British volume.

[85]  Henry Fuchs,et al.  Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms , 2002, Medical Image Anal..

[86]  Christian Winne,et al.  Overlay visualization in endoscopic ENT surgery , 2011, International Journal of Computer Assisted Radiology and Surgery.

[87]  Toshiya Nakaguchi 2-2 Projector-based Augmented Reality for Medicine , 2017 .

[88]  Jérôme Tonetti,et al.  Fluoroscopy-based navigation system in spine surgery , 2007, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[89]  Lejing Wang,et al.  First Deployments of Augmented Reality in Operating Rooms , 2012, Computer.

[90]  S. Weber,et al.  Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery. , 2013, The Journal of surgical research.

[91]  Takeo Kanade,et al.  An Image Overlay System for Medical Data Visualization , 1998, MICCAI.

[92]  Gabor Fichtinger,et al.  MRI image overlay: Application to arthrography needle insertion , 2007, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[93]  Fumio Kishino,et al.  Investigation of Errors in Perception of Stereoscopically Presented Virtual Object Locations in Real Display Space , 1994 .

[94]  Manabu Ito,et al.  A novel 3D guidance system using augmented reality for percutaneous vertebroplasty: technical note. , 2013, Journal of neurosurgery. Spine.

[95]  K. Chin,et al.  Real-time ultrasound-guided spinal anesthesia using the SonixGPS® needle tracking system: a case report , 2013, Canadian Journal of Anesthesia/Journal canadien d'anesthésie.

[96]  Guang-Zhong Yang,et al.  Intra-Operative Visualizations: Perceptual Fidelity and Human Factors , 2008, Journal of Display Technology.

[97]  Ichiro Sakuma,et al.  Augmented Reality Navigation With Automatic Marker-Free Image Registration Using 3-D Image Overlay for Dental Surgery , 2014, IEEE Transactions on Biomedical Engineering.

[98]  S. Botden,et al.  What is going on in augmented reality simulation in laparoscopic surgery? , 2008, Surgical Endoscopy.

[99]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[100]  Nassir Navab,et al.  Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection , 2008, IEEE Transactions on Medical Imaging.

[101]  R. Lederman,et al.  Robust automatic rigid registration of MRI and X-ray using external fiducial markers for XFM-guided interventional procedures. , 2010, Medical physics.

[102]  William E. Higgins,et al.  3D CT-Video Fusion for Image-Guided Bronchoscopy , 2008, Comput. Medical Imaging Graph..

[103]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[104]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[105]  Hongen Liao,et al.  3-D Augmented Reality for MRI-Guided Surgery Using Integral Videography Autostereoscopic Image Overlay , 2010, IEEE Transactions on Biomedical Engineering.

[106]  Wolfgang Birkfellner,et al.  Computer-enhanced stereoscopic vision in a head-mounted operating binocular. , 2003, Physics in medicine and biology.

[107]  Sebastian Bodenstedt,et al.  Context-aware Augmented Reality in laparoscopic surgery , 2013, Comput. Medical Imaging Graph..

[108]  Daniel Kendoff,et al.  Critical review of the current practice for computer-assisted navigation in total knee replacement surgery: cost-effectiveness and clinical outcome , 2011, Current reviews in musculoskeletal medicine.

[109]  Mark R. Wilson,et al.  Developing basic hand‐eye coordination skills for laparoscopic surgery using gaze training , 2010, BJU international.

[110]  R. Klatzky,et al.  Peripherally Inserted Central Catheter Placement With the Sonic Flashlight , 2009, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[111]  Raúl San José Estépar,et al.  Natural Orifice Transluminal Endoscopic Surgery (NOTES): An Opportunity for Augmented Reality Guidance , 2007, MMVR.

[112]  David Hawkes,et al.  Surface transparency makes stereo overlays unpredictable: the implications for augmented reality. , 2003, Studies in health technology and informatics.

[113]  Andrew D. Wiles,et al.  Virtual reality-enhanced ultrasound guidance: A novel technique for intracardiac interventions , 2008, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[114]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[115]  Nassir Navab,et al.  The Virtual Mirror: A New Interaction Paradigm for Augmented Reality Environments , 2009, IEEE Transactions on Medical Imaging.

[116]  K. Cleary,et al.  Image-Guided Procedures : A Review , 2006 .

[117]  D. Sindram,et al.  Real-time three-dimensional guided ultrasound targeting system for microwave ablation of liver tumours: a human pilot study. , 2011, HPB : the official journal of the International Hepato Pancreato Biliary Association.

[118]  F. Kishino,et al.  Cooperative work environment using virtual workspace , 1992, CSCW '92.

[119]  Barry S. Coller,et al.  Traversing the Valley of Death: A Guide to Assessing Prospects for Translational Success , 2009, Science Translational Medicine.

[120]  G Zheng,et al.  Reality-augmented virtual fluoroscopy for computer-assisted diaphyseal long bone fracture osteosynthesis: A novel technique and feasibility study results , 2008, Proceedings of the Institution of Mechanical Engineers. Part H, Journal of engineering in medicine.

[121]  A. Darzi,et al.  Augmented reality partial nephrectomy: examining the current status and future perspectives. , 2014, Urology.

[122]  R. Higuchi,et al.  Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery , 2010, Journal of hepato-biliary-pancreatic sciences.