Advanced Medical Displays: A Literature Review of Augmented Reality

The impressive development of medical imaging technology during the last decades provided physicians with an increasing amount of patient specific anatomical and functional data. In addition, the increasing use of non-ionizing real-time imaging, in particular ultrasound and optical imaging, during surgical procedures created the need for design and development of new visualization and display technology allowing physicians to take full advantage of rich sources of heterogeneous preoperative and intraoperative data. During 90's, medical augmented reality was proposed as a paradigm bringing new visualization and interaction solutions into perspective. This paper not only reviews the related literature but also establishes the relationship between subsets of this body of work in medical augmented reality. It finally discusses the remaining challenges for this young and active multidisciplinary research community.

[1]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[2]  J. M. Gerzso,et al.  Computer graphics and interactive techniques: 15th-17th July 1974. Boulder, Colorado, USA. Sponsored by the University of Colorado Computing Centre and ACM/SIGGRAPH , 1975, Comput. Aided Des..

[3]  P. Kelly,et al.  Computer-assisted stereotactic microsurgery for the treatment of intracranial neoplasms. , 1982, Neurosurgery.

[4]  S. Stellar.  Laser microsurgery for the treatment of intracranial neoplasms. , 1982, Neurosurgery.

[5]  J. Strohbehn,et al.  A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. , 1986, Journal of neurosurgery.

[6]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[7]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[8]  Marc Levoy,et al.  Display of surfaces from volume data , 1988, IEEE Computer Graphics and Applications.

[9]  E.M. Friets,et al.  A frameless stereotaxic operating microscope for neurosurgery , 1989, IEEE Transactions on Biomedical Engineering.

[10]  Howard Rheingold,et al.  Virtual Reality , 1991 .

[11]  T. Stoffregen,et al.  An ecological Theory of Motion Sickness and Postural Instability , 1991 .

[12]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[13]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[14]  Warren E. Smith,et al.  Correction of distortion in endoscope images , 1992, IEEE Trans. Medical Imaging.

[15]  FuchsHenry,et al.  Merging virtual objects with the real world , 1992 .

[16]  Wendy E. Mackay,et al.  Computer-Augmented Environments: Back to the Real World - Introduction to the Special Issue. , 1993 .

[17]  Wendy E. Mackay,et al.  Augmenting reality: adding computational dimensions to paper , 1993, CACM.

[18]  Steven K. Feiner,et al.  Knowledge-based augmented reality , 1993, CACM.

[19]  Wendy E. Mackay,et al.  Back to the real world , 1993, CACM.

[20]  William E. Lorensen,et al.  Enhancing reality in the operating room , 1993, Proceedings Visualization '93.

[21]  T M Peters,et al.  Integration of stereoscopic DSA and 3D MRI for image-guided neurosurgery. , 1994, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society.

[22]  Ravin Balakrishnan,et al.  Reaching for objects in VR displays: lag and frame rate , 1994, TCHI.

[23]  Reg G. Willson.  Modeling and calibration of automated zoom lenses , 1994, Other Conferences.

[24]  W. Eric L. Grimson,et al.  An automatic registration method for frameless stereotaxy, image guided surgery, and enhanced reality visualization , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[25]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[26]  David E. Breen,et al.  Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System , 1995, IEEE Trans. Vis. Comput. Graph..

[27]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[28]  Wright Dl,et al.  Using virtual reality to teach radiographic positioning. , 1995 .

[29]  J. Rolland,et al.  Using virtual reality to teach radiographic positioning. , 1995, Radiologic technology.

[30]  James E. Cutting,et al.  Chapter 3 – Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth* , 1995 .

[31]  David J. Hawkes,et al.  Neurosurgical Guidance Using the Stereo Microscope , 1995, CVRMed.

[32]  Paul Milgram,et al.  Perceptual issues in augmented reality , 1996, Electronic Imaging.

[33]  Mary C. Whitton,et al.  Technologies for augmented reality systems: realizing ultrasound-guided needle biopsies , 1996, SIGGRAPH.

[34]  Frederick P. Brooks,et al.  The computer scientist as toolsmith II , 1996, CACM.

[35]  Allen Van Gelder,et al.  Direct volume rendering with shading via three-dimensional textures , 1996, Proceedings of 1996 Symposium on Volume Visualization.

[36]  Mark A. Livingston,et al.  Managing latency in complex augmented reality systems , 1997, SI3D.

[37]  Richard L. Holloway,et al.  Registration Error Analysis for Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[38]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[39]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[40]  Robert J. Maciunas,et al.  Registration of head volume images using implantable fiducial markers , 1997, IEEE Transactions on Medical Imaging.

[41]  W. Lau,et al.  History of Endoscopic and Laparoscopic Surgery , 1997, World Journal of Surgery.

[42]  M. Murphy.  An automatic six-degree-of-freedom image registration algorithm for image-guided frameless stereotaxic radiosurgery. , 1997, Medical physics.

[43]  Frank Biocca,et al.  Virtual Eyes Can Rearrange Your Body: Adaptation to Visual Displacement in See-Through, Head-Mounted Displays , 1998, Presence.

[44]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[45]  Jay B. West,et al.  Predicting error in rigid-body point-based registration , 1998, IEEE Transactions on Medical Imaging.

[46]  Haiying Liu,et al.  Investigation of intraoperative brain deformation using a 1.5-T interventional MR system: preliminary results , 1998, IEEE Transactions on Medical Imaging.

[47]  Ivan Bricault,et al.  Registration of real and CT-derived virtual bronchoscopic images to assist transbronchial biopsy , 1998, IEEE Transactions on Medical Imaging.

[48]  M Scholz,et al.  Development of an endoscopic navigation system based on digital image processing. , 1998, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[49]  William M. Wells,et al.  Medical Image Computing and Computer-Assisted Intervention — MICCAI’98 , 1998, Lecture Notes in Computer Science.

[50]  A. Cuschieri,et al.  Task performance in endoscopic surgery is influenced by location of the image display. , 1998, Annals of surgery.

[51]  Yoshikazu Nakajima,et al.  Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization , 1998, IEEE Transactions on Medical Imaging.

[52]  Ramin Shahidi,et al.  Volumetric Image Guidance via a Stereotactic Endoscope , 1998, MICCAI.

[53]  M. Moran,et al.  New visualization techniques for in utero surgery: amnioscopy with a three-dimensional head-mounted display and a computer-controlled endoscope. , 1998, Journal of endourology.

[54]  Nassir Navab,et al.  Merging visible and invisible: two Camera-Augmented Mobile C-arm (CAMC) applications , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[55]  J. Wong,et al.  The use of active breathing control (ABC) to reduce margin for breathing motion. , 1999, International journal of radiation oncology, biology, physics.

[56]  Litynski Gs,et al.  Endoscopic Surgery: The History, the Pioneers , 1999, World Journal of Surgery.

[57]  R. P. Gaston,et al.  A System for Microscope-Assisted Guided Interventions , 2000, Stereotactic and Functional Neurosurgery.

[58]  J. Berger,et al.  Computer-vision-enabled augmented reality fundus biomicroscopy. , 1999, Ophthalmology.

[59]  Henry Fuchs,et al.  Optical Versus Video See-Through Head-Mounted Displays in Medical Visualization , 2000, Presence: Teleoperators & Virtual Environments.

[60]  Rebecca Fahrig,et al.  A Flouroscopic X-Ray Registration Process for Three-Dimensional Surgical Navigation , 2000, MICCAI.

[61]  George D. Stetten,et al.  Tomographic reflection to merge ultrasound images with direct vision , 2000, Proceedings 29th Applied Imagery Pattern Recognition Workshop.

[62]  G. Berci,et al.  History of endoscopy: what lessons have we learned from the past? , 2000, Surgical endoscopy.

[63]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Shyr Yu,et al.  Registration of physical space to laparoscopic image space for use in minimally invasive hepatic surgery , 2000, IEEE Transactions on Medical Imaging.

[65]  Tyrone L. Vincent,et al.  Analysis of Head Pose Accuracy in Augmented Reality , 2000, IEEE Trans. Vis. Comput. Graph..

[66]  Takeo Kanade,et al.  An Image Overlay System for Medical Data Visualization , 1998, MICCAI.

[67]  Ken Masamune,et al.  Three-Dimensional Slice Image Overlay System with Accurate Depth Perception for Surgery , 2000, MICCAI.

[68]  Yakup Genc,et al.  Augmented workspace: designing an AR testbed , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[69]  David J. Hawkes,et al.  Design and evaluation of a system for microscope-assisted guided interventions (MAGI) , 2000 .

[70]  Wolfgang Birkfellner,et al.  Development of the Varioscope AR. A see-through HMD for computer-aided surgery , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[71]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[72]  Larry S. Davis,et al.  Dynamic superimposition of synthetic objects on rigid and simple-deformable real objects , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[73]  K. Brock,et al.  Determination of ventilatory liver movement via radiographic evaluation of diaphragm position. , 2001, International journal of radiation oncology, biology, physics.

[74]  Hongen Liao,et al.  Intra-operative Real-Time 3-D Information Display System Based on Integral Videography , 2001, MICCAI.

[75]  Paul Suetens,et al.  A System to Support Laparoscopic Surgery by Augmented Reality Visualization , 2001, MICCAI.

[76]  Dimitris N. Metaxas,et al.  Methods for Modeling and Predicting Mechanical Deformations of the Breast Under External Perturbations , 2001, MICCAI.

[77]  Daisuke Deguchi,et al.  A Method for Tracking the Camera Motion of Real Endoscope by Epipolar Geometry Analysis and Virtual Endoscopy System , 2001, MICCAI.

[78]  Rasool Khadem,et al.  Endoscope Calibration and Accuracy Testing for 3D/2D Image Registration , 2001, MICCAI.

[79]  G. Stetten,et al.  Overlaying ultrasonographic images on direct vision. , 2001, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[80]  Masakatsu G. Fujie,et al.  A Novel Laser Guidance System for Alignment of Linear Surgical Tools: Its Principles and Performance Evaluation as a Man-Machine System , 2002, MICCAI.

[81]  Ève Coste-Manière,et al.  Flexible Calibration of Actuated Stereoscopic Endoscope for Overlay in Robot Assisted Surgery , 2002, MICCAI.

[82]  Yoshikazu Nakajima,et al.  3D Ultrasound System Using a Magneto-optic Hybrid Tracker for Augmented Reality Visualization in Laparoscopic Liver Surgery , 2002, MICCAI.

[83]  Terry M. Peters,et al.  Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: creating stereoscopic panoramas , 2002, IEEE Transactions on Medical Imaging.

[84]  Jay B. West,et al.  Implementation, calibration and accuracy testing of an image-enhanced endoscopy system , 2002, IEEE Transactions on Medical Imaging.

[85]  Blair MacIntyre,et al.  Estimating and adapting to registration errors in augmented reality systems , 2002, Proceedings IEEE Virtual Reality 2002.

[86]  Henry Fuchs,et al.  Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms , 2002, Medical Image Anal..

[87]  Ramin Shahidi,et al.  Validation of medical image processing in image-guided therapy , 2002, IEEE Transactions on Medical Imaging.

[88]  H. Iseki,et al.  Endoscopic Augmented Reality Navigation System for Endonasal Transsphenoidal Surgery to Treat Pituitary Tumors: Technical Note , 2002, Neurosurgery.

[89]  Ronald Azuma,et al.  Augmented-reality visualizations guided by cognition: perceptual heuristics for combining visible and obscured information , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[90]  Wolfgang Birkfellner,et al.  A head-mounted operating binocular for augmented reality visualization in medicine - design and initial evaluation , 2002, IEEE Transactions on Medical Imaging.

[91]  Rüdiger Westermann,et al.  Acceleration techniques for GPU-based volume rendering , 2003, IEEE Visualization, 2003. VIS 2003..

[92]  Bernhard Preim,et al.  Intraoperative augmented reality for minimally invasive liver interventions , 2003, SPIE Medical Imaging.

[93]  Ron Kikinis,et al.  A Navigation System for Augmenting Laparoscopic Ultrasound , 2003, MICCAI.

[94]  Heinrich Niemann,et al.  Robust Hand-Eye Calibration of an Endoscopic Surgery Robot Using Dual Quaternions , 2003, DAGM-Symposium.

[95]  Neil D. Glossop,et al.  Laser projection augmented reality system for computer-assisted surgery , 2003, CARS.

[96]  David Hawkes,et al.  Surface transparency makes stereo overlays unpredictable: the implications for augmented reality. , 2003, Studies in health technology and informatics.

[97]  Tobias Höllerer,et al.  Resolving multiple occluded layers in augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[98]  H. Niemann,et al.  Evaluation of computer-assisted image enhancement in minimal invasive endoscopic surgery. , 2004, Methods of information in medicine.

[99]  Luc Soler,et al.  An Accuracy Certified Augmented Reality System for Therapy Guidance , 2004, ECCV.

[100]  Hideki Yoshikawa,et al.  Development of a camera model and calibration procedure for oblique-viewing endoscopes , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[101]  R. Sader,et al.  Development of an Augmented Reality System for intraoperative navigation in maxillofacial surgery , 2004 .

[102]  Charles Baur,et al.  Hybrid method for both calibration and registration of an endoscope with an active optical tracker , 2004, CARS.

[103]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[104]  Stefan Wesarg,et al.  Accuracy of needle implantation in brachytherapy using a medical AR system: a phantom study , 2004, Medical Imaging: Image-Guided Procedures.

[105]  Nobuhiko Hata,et al.  Surgical navigation by autostereoscopic image overlay of integral videography , 2004, IEEE Transactions on Information Technology in Biomedicine.

[106]  Darius Burschka,et al.  Scale-Invariant Registration of Monocular Endoscopic Images to CT-Scans for Sinus Surgery , 2004, MICCAI.

[107]  Nassir Navab,et al.  Synchronizing 3D movements for quantitative comparison and simultaneous visualization of actions , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[108]  R. Thoranaghatte,et al.  Endoscope-based hybrid navigation system for minimally invasive ventral spine surgeries , 2005, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[109]  Daisuke Deguchi,et al.  Hybrid Bronchoscope Tracking Using a Magnetic Tracking Sensor and Image Registration , 2005, MICCAI.

[110]  Wolfgang Birkfellner,et al.  A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus , 2005, IEEE Transactions on Medical Imaging.

[111]  Luc Soler,et al.  A low cost and accurate guidance system for laparoscopic surgery: validation on an abdominal phantom , 2005, VRST '05.

[112]  Ken Masamune,et al.  Image overlay guidance for needle insertion in CT scanner , 2005, IEEE Transactions on Biomedical Engineering.

[113]  Philippe C. Cattin,et al.  Camera-marker alignment framework and comparison with hand-eye calibration for augmented reality applications , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[114]  Darius Burschka,et al.  DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability , 2005, MICCAI.

[115]  Guang-Zhong Yang,et al.  Laparoscope Self-calibration for Robotic Assisted Minimally Invasive Surgery , 2005, MICCAI.

[116]  Nassir Navab,et al.  Automatic Patient Registration for Port Placement in Minimally Invasixe Endoscopic Surgery , 2005, MICCAI.

[117]  Nassir Navab,et al.  Recovery of Surgical Workflow Without Explicit Models , 2006, MICCAI.

[118]  Philippe C. Cattin,et al.  Markerless Endoscopic Registration and Referencing , 2006, MICCAI.

[119]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[120]  Nassir Navab,et al.  Navigated Three Dimensional Beta Probe for Optimal Cancer Resection , 2006, MICCAI.

[121]  Dieter Schmalstieg,et al.  Liver Surgery Planning Using Virtual Reality , 2006, IEEE Computer Graphics and Applications.

[122]  J. Rolland,et al.  Head-worn displays: a review , 2006, Journal of Display Technology.

[123]  Peter Anderson,et al.  Method for estimating dynamic EM tracking accuracy of surgical navigation tools , 2006, SPIE Medical Imaging.

[124]  Ekkehard Euler,et al.  Hybrid Navigation Interface for Orthopedic and Trauma Surgery , 2006, MICCAI.

[125]  Gang Luo,et al.  Use of an augmented-vision device for visual search by patients with tunnel vision. , 2006, Investigative ophthalmology & visual science.

[126]  Hans-Peter Meinzer,et al.  Development of a navigation system for endoluminal brachytherapy in human lungs , 2006, SPIE Medical Imaging.

[127]  Yasuo Ito,et al.  Clinical study for registration-free 3D-navigation with the SIREMOBIL Iso-C3D mobile C-arm , 2006 .

[128]  Nassir Navab,et al.  Online Estimation of the Target Registration Error for n -Ocular Optical Tracking Systems , 2007, MICCAI.

[129]  W. Thumfart,et al.  Image-guided endoscopic ENT surgery , 2007, European Archives of Oto-Rhino-Laryngology.

[130]  Nassir Navab,et al.  Measurement of absolute latency for video see through augmented reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[131]  Nassir Navab,et al.  Magneto-Optic Tracking of a Flexible Laparoscopic Ultrasound Transducer for Laparoscope Augmentation , 2007, MICCAI.

[132]  Dieter Schmalstieg,et al.  Interactive Focus and Context Visualization for Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[133]  Hans-Peter Meinzer,et al.  Soft tissue navigation for laparoscopic prostatectomy: evaluation of camera pose estimation for enhanced visualization , 2007, SPIE Medical Imaging.

[134]  Nassir Navab,et al.  Registration-free laparoscope augmentation for intra-operative liver resection planning , 2007, SPIE Medical Imaging.

[135]  Nassir Navab,et al.  Laparoscopic Virtual Mirror New Interaction Paradigm for Monitor Based Augmented Reality , 2007, 2007 IEEE Virtual Reality Conference.

[136]  Paul Suetens,et al.  Evaluation of a Novel Calibration Technique for Optically Tracked Oblique Laparoscopes , 2007, MICCAI.

[137]  Nassir Navab,et al.  Fiducial-Free Registration Procedure for Navigated Bronchoscopy , 2007, MICCAI.

[138]  Nassir Navab,et al.  Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[139]  Guang-Zhong Yang,et al.  pq-space Based Non-Photorealistic Rendering for Augmented Reality , 2007, MICCAI.

[140]  Gabor Fichtinger,et al.  MRI image overlay: Application to arthrography needle insertion , 2007, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[141]  M. Feuerstein,et al.  Navigation in endoscopic soft tissue surgery: perspectives and limitations. , 2008, Journal of endourology.

[142]  Nassir Navab,et al.  Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection , 2008, IEEE Transactions on Medical Imaging.

[143]  Dima Damen,et al.  Recognizing linked events: Searching the space of feasible explanations , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.