The status of augmented reality in laparoscopic surgery as of 2016

This article establishes a comprehensive review of all the different methods proposed by the literature concerning augmented reality in intra-abdominal minimally invasive surgery (also known as laparoscopic surgery). A solid background of surgical augmented reality is first provided in order to support the survey. Then, the various methods of laparoscopic augmented reality as well as their key tasks are categorized in order to better grasp the current landscape of the field. Finally, the various issues gathered from these reviewed approaches are organized in order to outline the remaining challenges of augmented reality in laparoscopic surgery.

[1]  Sebastian Bodenstedt,et al.  Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. , 2012, Medical physics.

[2]  A. Park,et al.  Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography , 2010, Surgical Endoscopy.

[3]  Lena Maier-Hein,et al.  Real-Time Range Imaging in Health Care: A Survey , 2013, Time-of-Flight and Depth Imaging.

[4]  Heinz-Otto Peitgen,et al.  Illustrative visualization of 3D planning models for augmented reality in liver surgery , 2010, International Journal of Computer Assisted Radiology and Surgery.

[5]  Sukryool Kang,et al.  Application of single-image camera calibration for ultrasound augmented laparoscopic visualization , 2015, Medical Imaging.

[6]  Nassir Navab,et al.  Intraoperative Laparoscope Augmentation for Port Placement and Resection Planning in Minimally Invasive Liver Resection , 2008, IEEE Transactions on Medical Imaging.

[7]  Guang-Zhong Yang,et al.  Enhanced visualisation for minimally invasive surgery , 2012, International Journal of Computer Assisted Radiology and Surgery.

[8]  C. Hillenbrand,et al.  Evaluation of respiratory liver and kidney movements for MRI navigator gating , 2011, Journal of magnetic resonance imaging : JMRI.

[9]  Charles Baur,et al.  Real-time update of 3D deformable models for computer aided liver surgery , 2008, 2008 19th International Conference on Pattern Recognition.

[10]  Armin Besirevic,et al.  Laparoscopic Navigated Liver Resection: Technical Aspects and Clinical Practice in Benign Liver Tumors , 2012, Case reports in surgery.

[11]  J. Marescaux,et al.  Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy , 2015, Langenbeck's Archives of Surgery.

[12]  S. Weber,et al.  Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery. , 2013, The Journal of surgical research.

[13]  Jin U Kang,et al.  Direct three-dimensional ultrasound-to-video registration using photoacoustic markers , 2013, Journal of biomedical optics.

[14]  Luc Soler,et al.  Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery , 2016, Medical Image Anal..

[15]  J. L. Moyano-Cuevas,et al.  Anatomical changes due to pneumoperitoneum analyzed by MRI: an experimental study in pigs , 2011, Surgical and Radiologic Anatomy.

[16]  Gabor Fichtinger,et al.  Augmented reality visualization using Image-Overlay for MR-guided interventions: system description, feasibility, and initial evaluation in a spine phantom. , 2011, AJR. American journal of roentgenology.

[17]  João Pedro Barreto,et al.  A New Solution for Camera Calibration and Real-Time Image Distortion Correction in Medical Endoscopy–Initial Technical Evaluation , 2012, IEEE Transactions on Biomedical Engineering.

[18]  Paul Suetens,et al.  Evaluation of a Novel Calibration Technique for Optically Tracked Oblique Laparoscopes , 2007, MICCAI.

[19]  Michael C. Yip,et al.  Tissue Tracking and Registration for Image-Guided Surgery , 2012, IEEE Transactions on Medical Imaging.

[20]  Luc Soler,et al.  Using Shading to Register an Intraoperative CT Scan to a Laparoscopic Image , 2015, CARE@MICCAI.

[21]  A. Darzi,et al.  Perception and orientation in minimally invasive surgery. , 2012, Archives of surgery.

[22]  K. Schaller,et al.  Augmented reality-assisted bypass surgery: embracing minimal invasiveness. , 2015, World neurosurgery.

[23]  Caroline Petitjean,et al.  A review of 3 D / 2 D registration methods for image-guided interventions , 2016 .

[24]  Ryutarou Ohbuchi,et al.  Merging virtual objects with the real world: seeing ultrasound imagery within the patient , 1992, SIGGRAPH.

[25]  Lutz Ritter,et al.  A paradigm shift in orthognathic surgery? A comparison of navigation, computer-aided designed/computer-aided manufactured splints, and "classic" intermaxillary splints to surgical transfer of virtual orthognathic planning. , 2013, Journal of oral and maxillofacial surgery : official journal of the American Association of Oral and Maxillofacial Surgeons.

[26]  Tessa Buckle,et al.  Multispectral Fluorescence Imaging During Robot-assisted Laparoscopic Sentinel Node Biopsy: A First Step Towards a Fluorescence-based Anatomic Roadmap. , 2017, European urology.

[27]  Li Zhang,et al.  Flat-Panel CT and the Future of OR Imaging and Navigation , 2015 .

[28]  Luc Soler,et al.  Fast 3D Structure From Motion with Missing Points from Registration of Partial Reconstructions , 2012, AMDO.

[29]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[30]  Eigil Samset,et al.  Augmented Reality for Minimally Invasive Surgery: Overview and Some Recent Advances , 2010 .

[31]  O. Ratib,et al.  Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion , 2011, Journal of hepato-biliary-pancreatic sciences.

[32]  Armin Schneider,et al.  Time-of-Flight 3-D Endoscopy , 2009, MICCAI.

[33]  David J. Hawkes,et al.  Progressive internal landmark registration for surgical navigation in laparoscopic gastrectomy for gastric cancer , 2016, International Journal of Computer Assisted Radiology and Surgery.

[34]  Terry M. Peters,et al.  Registration of 3D shapes under anisotropic scaling , 2015, International Journal of Computer Assisted Radiology and Surgery.

[35]  Guang-Zhong Yang,et al.  Invisible Shadow for Navigation and Planning in Minimal Invasive Surgery , 2005, MICCAI.

[36]  Guang-Zhong Yang,et al.  Motion Compensated SLAM for Image Guided Surgery , 2010, MICCAI.

[37]  João Pedro Barreto,et al.  Continuous Zoom Calibration by Tracking Salient Points in Endoscopic Video , 2014, MICCAI.

[38]  Ken Masamune,et al.  Image overlay guidance for needle insertion in CT scanner , 2005, IEEE Transactions on Biomedical Engineering.

[39]  Martin Styner,et al.  Comparison and Evaluation of Methods for Liver Segmentation From CT Datasets , 2009, IEEE Transactions on Medical Imaging.

[40]  Luc Soler,et al.  Automatic detection of endoscope in intraoperative CT image: Application to AR guidance in laparoscopic surgery , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[41]  Darius Burschka,et al.  DaVinci Canvas: A Telerobotic Surgical System with Integrated, Robot-Assisted, Laparoscopic Ultrasound Capability , 2005, MICCAI.

[42]  Gregory Hager,et al.  Vision-based navigation in image-guided interventions. , 2011, Annual review of biomedical engineering.

[43]  A. Bartoli,et al.  Computer assisted minimally invasive surgery: is medical computer vision the answer to improving laparosurgery? , 2012, Medical hypotheses.

[44]  Frank Sauer,et al.  Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation , 2006, International Journal of Computer Vision.

[45]  Makoto Hashizume,et al.  An augmented reality navigation system for pediatric oncologic surgery based on preoperative CT and MRI images. , 2013, Journal of pediatric surgery.

[46]  Danail Stoyanov,et al.  Robust Real-Time Visual Odometry for Stereo Endoscopy Using Dense Quadrifocal Tracking , 2014, IPCAI.

[47]  I. Gill,et al.  Current progress on augmented reality visualization in endoscopic surgery , 2012, Current opinion in urology.

[48]  Nassir Navab,et al.  The Virtual Mirror: A New Interaction Paradigm for Augmented Reality Environments , 2009, IEEE Transactions on Medical Imaging.

[49]  Gregory D. Hager,et al.  Articulated object tracking by rendering consistent appearance parts , 2009, 2009 IEEE International Conference on Robotics and Automation.

[50]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[51]  Nassir Navab,et al.  Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[52]  V. Agnus,et al.  Context-specific selection of algorithms for recursive feature tracking in endoscopic image using a new methodology , 2015, Comput. Medical Imaging Graph..

[53]  D. Stoyanov,et al.  Intraoperative measurement of bowel oxygen saturation using a multispectral imaging laparoscope. , 2015, Biomedical optics express.

[54]  Su-Lin Lee,et al.  From medical images to minimally invasive intervention: Computer assistance for robotic surgery , 2010, Comput. Medical Imaging Graph..

[55]  Jean-Denis Durou,et al.  Numerical methods for shape-from-shading: A new survey with benchmarks , 2008, Comput. Vis. Image Underst..

[56]  James E. Cutting,et al.  Chapter 3 – Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth* , 1995 .

[57]  Christian Winne,et al.  Overlay visualization in endoscopic ENT surgery , 2011, International Journal of Computer Assisted Radiology and Surgery.

[58]  Terry M. Peters,et al.  Fused Video and Ultrasound Images for Minimally Invasive Partial Nephrectomy: A Phantom Study , 2010, MICCAI.

[59]  B. Alaraimi,et al.  A Randomized Prospective Study Comparing Acquisition of Laparoscopic Skills in Three-Dimensional (3D) vs. Two-Dimensional (2D) Laparoscopy , 2014, World Journal of Surgery.

[60]  Adrien Bartoli,et al.  Tracking by Detection for Interactive Image Augmentation in Laparoscopy , 2012, WBIR.

[61]  Stefanie Speidel,et al.  Automatic classification of minimally invasive instruments based on endoscopic image sequences , 2009, Medical Imaging.

[62]  S. Nitzsche,et al.  Cardio navigation: planning, simulation, and augmented reality in robotic assisted endoscopic bypass grafting. , 2005, The Annals of thoracic surgery.

[63]  Salvatore Micali,et al.  New trends in minimally invasive urological surgery. What is beyond the robot? , 2013, World Journal of Urology.

[64]  Luc Soler,et al.  Biomechanically Driven Registration of Pre- to Intra-Operative 3D Images for Laparoscopic Surgery , 2013, MICCAI.

[65]  Oliver Kutter,et al.  CAMPAR: a software framework guaranteeing quality for medical augmented reality , 2006 .

[66]  Adrien Bartoli,et al.  3D Reconstruction in Laparoscopy with Close-Range Photometric Stereo , 2012, MICCAI.

[67]  Ghassan Hamarneh,et al.  Uncertainty-Encoded Augmented Reality for Robot-Assisted Partial Nephrectomy: A Phantom Study , 2013, AE-CAI.

[68]  David J. Hawkes,et al.  Fast Semi-dense Surface Reconstruction from Stereoscopic Video in Laparoscopic Surgery , 2014, IPCAI.

[69]  Luc Soler,et al.  Enhanced-Reality Video Fluorescence: A Real-Time Assessment of Intestinal Viability , 2014, Annals of surgery.

[70]  Philip J. Edwards,et al.  Depth perception of stereo overlays in image-guided surgery , 2004, SPIE Medical Imaging.

[71]  Luc Soler,et al.  A low cost and accurate guidance system for laparoscopic surgery: validation on an abdominal phantom , 2005, VRST '05.

[72]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[73]  Russell H. Taylor,et al.  Robot-Assisted Laparoscopic Ultrasound , 2010, IPCAI.

[74]  Simon Weidert,et al.  Machine learning-based augmented reality for improved surgical scene understanding , 2015, Comput. Medical Imaging Graph..

[75]  Hongen Liao,et al.  3-D Augmented Reality for MRI-Guided Surgery Using Integral Videography Autostereoscopic Image Overlay , 2010, IEEE Transactions on Biomedical Engineering.

[76]  Brice Gayet,et al.  Augmented Reality Navigation Surgery Facilitates Laparoscopic Rescue of Failed Portal Vein Embolization. , 2016, Journal of the American College of Surgeons.

[77]  R. Higuchi,et al.  Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery , 2010, Journal of hepato-biliary-pancreatic sciences.

[78]  Lena Maier-Hein,et al.  3D surface reconstruction for laparoscopic computer-assisted interventions: comparison of state-of-the-art methods , 2011, Medical Imaging.

[79]  Ara Darzi,et al.  The current and future use of imaging in urological robotic surgery: a survey of the European Association of Robotic Urological Surgeons , 2015, The international journal of medical robotics + computer assisted surgery : MRCAS.

[80]  Qi Zhang,et al.  Volume Visualization: A Technical Overview with a Focus on Medical Applications , 2011, Journal of Digital Imaging.

[81]  Carl-Fredrik Westin,et al.  Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures , 2009, International Journal of Computer Assisted Radiology and Surgery.

[82]  Jason Geng,et al.  Review of 3-D Endoscopic Surface Imaging Techniques , 2014, IEEE Sensors Journal.

[83]  M. Feuerstein,et al.  Navigation in endoscopic soft tissue surgery: perspectives and limitations. , 2008, Journal of endourology.

[84]  Hans-Peter Meinzer,et al.  Statistical shape models for 3D medical image segmentation: A review , 2009, Medical Image Anal..

[85]  G. Andriole,et al.  2D versus 3D visualization: impact on laparoscopic proficiency using the fundamentals of laparoscopic surgery skill set. , 2012, Journal of laparoendoscopic & advanced surgical techniques. Part A.

[86]  Nazim Haouchine,et al.  Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery , 2015, IEEE Transactions on Visualization and Computer Graphics.

[87]  Ken Masamune,et al.  Non-metal Slice Image Overlay Display System Used Inside the Open Type MRI , 2008, MIAR.

[88]  Anders Ynnerman,et al.  Local Histograms for Design of Transfer Functions in Direct Volume Rendering , 2006, IEEE Transactions on Visualization and Computer Graphics.

[89]  Robert L. Galloway,et al.  Compensating for intraoperative soft-tissue deformations using incomplete surface data and finite elements , 2005, IEEE Transactions on Medical Imaging.

[90]  Daniel Pizarro-Perez,et al.  Computer-Assisted Laparoscopic myomectomy by augmenting the uterus with pre-operative MRI data , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[91]  Luc Soler,et al.  On-patient see-through augmented reality based on visual SLAM , 2016, International Journal of Computer Assisted Radiology and Surgery.

[92]  Sukryool Kang,et al.  On-demand calibration and evaluation for electromagnetically tracked laparoscope in augmented reality visualization , 2016, International Journal of Computer Assisted Radiology and Surgery.

[93]  Daniel J. Thomas,et al.  Augmented reality in surgery: The Computer-Aided Medicine revolution. , 2016, International journal of surgery.

[94]  Daniel Pizarro-Perez,et al.  Realtime Wide-Baseline Registration of the Uterus in Laparoscopic Videos Using Multiple Texture Maps , 2013, AE-CAI.

[95]  Nazim Haouchine,et al.  Automatic Alignment of Pre and Intraoperative Data Using Anatomical Landmarks for Augmented Laparoscopic Liver Surgery , 2014, ISBMS.

[96]  Nima Najmaei,et al.  Image‐guided techniques in renal and hepatic interventions , 2013, The international journal of medical robotics + computer assisted surgery : MRCAS.

[97]  Andriy Myronenko,et al.  Point Set Registration: Coherent Point Drift , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[98]  Ma Conway,et al.  Handbook of perception and cognition , 1996 .

[99]  Zahra Ronaghi,et al.  Toward real-time remote processing of laparoscopic video , 2015, Journal of medical imaging.

[100]  Jay B. West,et al.  Implementation, calibration and accuracy testing of an image-enhanced endoscopy system , 2002, IEEE Transactions on Medical Imaging.

[101]  J. Marescaux,et al.  Augmented-reality-assisted laparoscopic adrenalectomy. , 2004, JAMA.

[102]  Thomas Wittenberg,et al.  Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods , 2016, IEEE Journal of Biomedical and Health Informatics.

[103]  R. Galloway,et al.  Kidney deformation and intraprocedural registration: a study of elements of image-guided kidney surgery. , 2011, Journal of endourology.

[104]  P. Kelly,et al.  Computer-assisted stereotaxic laser resection of intra-axial brain neoplasms. , 1986, Journal of neurosurgery.

[105]  Ghassan Hamarneh,et al.  Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors , 2016, International Journal of Computer Assisted Radiology and Surgery.

[106]  Dan Wang,et al.  Real Time 3D Visualization of Intraoperative Organ Deformations Using Structured Dictionary , 2012, IEEE Transactions on Medical Imaging.

[107]  D. Louis Collins,et al.  DVV: A Taxonomy for Mixed Reality Visualization in Image Guided Surgery , 2012, IEEE Transactions on Visualization and Computer Graphics.

[108]  Lena Maier-Hein,et al.  Comparative Validation of Single-Shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[109]  David J. Hawkes,et al.  Design and evaluation of a system for microscope-assisted guided interventions (MAGI) , 1999, IEEE Transactions on Medical Imaging.

[110]  J. Shuhaiber,et al.  Augmented reality in surgery. , 2004, Archives of surgery.

[111]  David J. Hawkes,et al.  Design and evaluation of a system for microscope-assisted guided interventions (MAGI) , 2000 .

[112]  Yuichiro Hayashi,et al.  Development and clinical application of surgical navigation system for laparoscopic hepatectomy , 2015, Medical Imaging.

[113]  Daniel Pizarro-Perez,et al.  Feature-Based Deformable Surface Detection with Self-Occlusion Reasoning , 2011, International Journal of Computer Vision.

[114]  T. Peters.  Image-guidance for surgical procedures , 2006, Physics in medicine and biology.

[115]  Nassir Navab,et al.  Magneto-Optical Tracking of Flexible Laparoscopic Ultrasound: Model-Based Online Detection and Correction of Magnetic Tracking Errors , 2009, IEEE Transactions on Medical Imaging.

[116]  D. Bartz,et al.  Visualization and Exploration of Segmented Anatomic Structures , 2010 .

[117]  Andrew D. Wiles,et al.  Accuracy assessment and interpretation for optical tracking systems , 2004, Medical Imaging: Image-Guided Procedures.

[118]  Wolfgang Birkfellner,et al.  Computer-enhanced stereoscopic vision in a head-mounted operating binocular. , 2003, Physics in medicine and biology.

[119]  Russell H. Taylor,et al.  Concurrent photoacoustic markers for direct three-dimensional ultrasound to video registration , 2014, Photonics West - Biomedical Optics.

[120]  Philippe C. Cattin,et al.  Markerless Endoscopic Registration and Referencing , 2006, MICCAI.

[121]  J. Strohbehn,et al.  A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. , 1986, Journal of neurosurgery.

[122]  Inderbir S. Gill,et al.  Augmented Reality for Image-Guided Surgery in Urology , 2010 .

[123]  I. Gill,et al.  Imaging-assisted endoscopic surgery: Cleveland Clinic experience. , 2008, Journal of endourology.

[124]  Kazuhiro Araki,et al.  Surgical navigation using three-dimensional computed tomography images fused intraoperatively with live video. , 2010, Journal of endourology.

[125]  Luc Soler,et al.  Real-time 3D image reconstruction guidance in liver resection surgery. , 2014, Hepatobiliary surgery and nutrition.

[126]  Adrien E. Desjardins,et al.  Interventional multi-spectral photoacoustic imaging in laparoscopic surgery , 2016, SPIE BiOS.

[127]  Guang-Zhong Yang,et al.  Robust ultrasound probe tracking: initial clinical experiences during robot-assisted partial nephrectomy , 2015, International Journal of Computer Assisted Radiology and Surgery.

[128]  Marco Caversaccio,et al.  Image-Guided Otorhinolaryngology , 2014 .

[129]  Hideki Yoshikawa,et al.  Development of a camera model and calibration procedure for oblique-viewing endoscopes , 2004, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[130]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[131]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[132]  Guang-Zhong Yang,et al.  Comparative effectiveness and safety of image guidance systems in neurosurgery: a preclinical randomized study. , 2015, Journal of neurosurgery.

[133]  Nazim Haouchine,et al.  Using contours as boundary conditions for elastic registration during minimally invasive hepatic surgery , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[134]  Fabrizio Cutolo,et al.  Application of a New Wearable Augmented Reality Video See-Through Display to Aid Percutaneous Procedures in Spine Surgery , 2016, AVR.

[135]  Marco Caversaccio,et al.  Image-guided surgical microscope with mounted mini-tracker , 2004, CARS.

[136]  Nazim Haouchine,et al.  Surgical Augmented Reality with Topological Changes , 2015, MICCAI.

[137]  Logan W. Clements,et al.  Feasibility study for image-guided kidney surgery: assessment of required intraoperative surface for accurate physical to image space registrations. , 2008, Medical physics.

[138]  Guang-Zhong Yang,et al.  Soft tissue deformation tracking for robotic assisted minimally invasive surgery , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[139]  Lena Maier-Hein,et al.  Crowdsourcing for Reference Correspondence Generation in Endoscopic Images , 2014, MICCAI.

[140]  Ziv Yaniv,et al.  Towards a Clinical Stereoscopic Augmented Reality System for Laparoscopic Surgery , 2013, CLIP.

[141]  Michael I. Miga,et al.  Model-Assisted Image-Guided Liver Surgery Using Sparse Intraoperative Data , 2012 .

[142]  O. Muensterer,et al.  Google Glass in pediatric surgery: an exploratory study. , 2014, International journal of surgery.

[143]  Ève Coste-Manière,et al.  Interactive Guidance by Image Overlay in Robot Assisted Coronary Artery Bypass , 2003, MICCAI.

[144]  H. Meinzer,et al.  Augmented reality visualization during laparoscopic radical prostatectomy. , 2011, Journal of endourology.

[145]  Erlend Fagertun Hofstad,et al.  Navigated laparoscopy – liver shift and deformation due to pneumoperitoneum in an animal model , 2012, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[146]  Anwar Abdalbari,et al.  Endoscopy-MR Image Fusion for Image Guided Procedures , 2013, Int. J. Biomed. Imaging.

[147]  Francisco Chinesta,et al.  Computational vademecums for real‐time simulation of surgical cutting in haptic environments , 2016 .

[148]  Lena Maier-Hein,et al.  ToF Meets RGB: Novel Multi-Sensor Super-Resolution for Hybrid 3-D Endoscopy , 2013, MICCAI.

[149]  Sebastian Bodenstedt,et al.  Context-aware Augmented Reality in laparoscopic surgery , 2013, Comput. Medical Imaging Graph..

[150]  G. Abba,et al.  A practical multi-plane method for a low-cost camera calibration technique , 1999, 1999 European Control Conference (ECC).

[151]  Makoto Hashizume,et al.  Intraoperative Magnetic Tracker Calibration Using a Magneto-Optic Hybrid Tracker for 3-D Ultrasound-Based Navigation in Laparoscopic Surgery , 2008, IEEE Transactions on Medical Imaging.

[152]  A. Darzi,et al.  Augmented reality partial nephrectomy: examining the current status and future perspectives. , 2014, Urology.

[153]  Naoki Suzuki,et al.  Identification of inferior pancreaticoduodenal artery during pancreaticoduodenectomy using augmented reality‐based navigation system , 2014, Journal of hepato-biliary-pancreatic sciences.

[154]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[155]  Adrien Bartoli,et al.  Template-Based Conformal Shape-from-Motion from Registered Laparoscopic Images , 2011, MIUA.

[156]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[157]  Nassir Navab,et al.  Personalized, relevance-based Multimodal Robotic Imaging and augmented reality for Computer Assisted Interventions , 2016, Medical Image Anal..

[158]  Guang-Zhong Yang,et al.  An effective visualisation and registration system for image-guided robotic partial nephrectomy , 2012, Journal of Robotic Surgery.

[159]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[160]  Nassir Navab,et al.  Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention , 2008, Medical Image Anal..

[161]  Naoki Suzuki,et al.  Clinical application of navigation surgery using augmented reality in the abdominal field , 2015, Surgery Today.

[162]  Nassir Navab,et al.  Depth Perception - A Major Issue in Medical AR: Evaluation Study by Twenty Surgeons , 2006, MICCAI.

[163]  K. Cleary,et al.  Image-guided interventions: technology review and clinical applications. , 2010, Annual review of biomedical engineering.

[164]  Peter Yellowlees,et al.  3-D telestration: a teaching tool for robotic surgery. , 2008, Journal of laparoendoscopic & advanced surgical techniques. Part A.

[165]  Ghassan Hamarneh,et al.  A Survey on Shape Correspondence , 2011, Comput. Graph. Forum.

[166]  Dieter Schmalstieg,et al.  Interactive Focus and Context Visualization for Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[167]  Pieter T. Eendebak,et al.  Improved depth perception with three-dimensional auxiliary display and computer generated three-dimensional panoramic overviews in robot-assisted laparoscopy , 2014, Journal of medical imaging.

[168]  Adrien Bartoli,et al.  Shape-from-Polarization in laparoscopy , 2013, 2013 IEEE 10th International Symposium on Biomedical Imaging.

[169]  J. Marescaux,et al.  Totally Endoscopic Magnetic Enteral Bypass by External Guided Rendez-Vous Technique , 2011, Surgical innovation.

[170]  Jeffrey A. Cadeddu,et al.  Toward Long-Term and Accurate Augmented-Reality for Monocular Endoscopic Videos , 2014, IEEE Transactions on Biomedical Engineering.

[171]  Alexandre Hostettler,et al.  Registration of Preoperative Liver Model for Laparoscopic Surgery from Intraoperative 3D Acquisition , 2013, AE-CAI.

[172]  Philippe Poignet,et al.  3D soft-tissue tracking using spatial-color joint probability distribution and thin-plate spline model , 2014, Pattern Recognit..

[173]  Lena Maier-Hein,et al.  Tissue classification for laparoscopic image understanding based on multispectral texture analysis , 2017, Journal of medical imaging.

[174]  Nima Najmaei,et al.  Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging – techniques and applications , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[175]  Ghassan Hamarneh,et al.  Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video , 2015, Medical Image Anal..

[176]  Adrien Bartoli,et al.  Combining Conformal Deformation and Cook–Torrance Shading for 3-D Reconstruction in Laparoscopy , 2014, IEEE Transactions on Biomedical Engineering.

[177]  Julien Nauroy,et al.  Real time planning, guidance and validation of surgical acts using 3D segmentations, augmented reality projections and surgical tools video tracking , 2010, Medical Imaging.

[178]  Christophe Doignon,et al.  A structured light-based laparoscope with real-time organs' surface reconstruction for minimally invasive surgery , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[179]  Yen-Wei Chen,et al.  A scope cylinder rotation tracking method for oblique-viewing endoscopes without attached sensing device , 2010, The 2nd International Conference on Software Engineering and Data Mining.

[180]  Erlend Fagertun Hofstad,et al.  Liver deformation in an animal model due to pneumoperitoneum assessed by a vessel-based deformable registration , 2014, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[181]  Yan Zhang,et al.  Tissue classification for laparoscopic image understanding based on multispectral texture analysis , 2016, SPIE Medical Imaging.

[182]  Gian Luca Mariottini,et al.  A Fast and Accurate Feature-Matching Algorithm for Minimally-Invasive Endoscopic Images , 2013, IEEE Transactions on Medical Imaging.

[183]  Makoto Hashizume,et al.  Image-guided laparoscopic surgery in an open MRI operating theater , 2013, Surgical Endoscopy.

[184]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[185]  Paolo Dario,et al.  EndoCAS navigator platform: a common platform for computer and robotic assistance in minimally invasive surgery , 2008, The international journal of medical robotics + computer assisted surgery : MRCAS.

[186]  Simon K. Warfield,et al.  Abdominal Images Non-rigid Registration Using Local-Affine Diffeomorphic Demons , 2011, Abdominal Imaging.

[187]  J. M. M. Montiel,et al.  Visual SLAM for Handheld Monocular Endoscope , 2014, IEEE Transactions on Medical Imaging.

[188]  Ivan Viola,et al.  Automatic Transfer Functions Based on Informational Divergence , 2011, IEEE Transactions on Visualization and Computer Graphics.

[189]  Jonathan M Sorger,et al.  Augmented reality and cone beam CT guidance for transoral robotic surgery , 2015, Journal of Robotic Surgery.

[190]  Luc Soler,et al.  Augmented Reality-Guided Artery-First Pancreatico-Duodenectomy , 2013, Journal of Gastrointestinal Surgery.

[191]  Jerry L Prince,et al.  Current methods in medical image segmentation. , 2000, Annual review of biomedical engineering.

[192]  Guang-Zhong Yang,et al.  Pico Lantern: Surface reconstruction and augmented reality in laparoscopic surgery using a pick-up laser projector , 2015, Medical Image Anal..

[193]  Luc Soler,et al.  Real-time navigation by fluorescence-based enhanced reality for precise estimation of future anastomotic site in digestive surgery , 2014, Surgical Endoscopy.

[194]  Stephan Reichelt,et al.  Depth cues in human visual perception and their realization in 3D displays , 2010, Defense + Commercial Sensing.

[195]  Guang-Zhong Yang,et al.  Probabilistic Tracking of Affine-Invariant Anisotropic Regions , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[196]  Gary K. L. Tam,et al.  Registration of 3D Point Clouds and Meshes: A Survey from Rigid to Nonrigid , 2013, IEEE Transactions on Visualization and Computer Graphics.

[197]  Beat P. Müller-Stich,et al.  Soft tissue navigation for laparoscopic partial nephrectomy , 2008, International Journal of Computer Assisted Radiology and Surgery.

[198]  J. Kaspersen,et al.  Laparoscopic navigation pointer for three-dimensional image–guided surgery , 2004, Surgical Endoscopy And Other Interventional Techniques.

[199]  Toshiya Nakaguchi,et al.  Ultrasound Image Overlay onto Endoscopic Image by Fusing 2D-3D Tracking of Laparoscopic Ultrasound Probe , 2014, AE-CAI.

[200]  Sébastien Ourselin,et al.  2D-3D Pose Tracking of Rigid Instruments in Minimally Invasive Surgery , 2014, IPCAI.

[201]  Philippe Zanne,et al.  Stitching Planning in Laparoscopic Surgery: Towards Robot-assisted Suturing , 2009, Int. J. Robotics Res..

[202]  Stefan Weber,et al.  Evaluation of a portable image overlay projector for the visualisation of surgical navigation data: phantom studies , 2012, International Journal of Computer Assisted Radiology and Surgery.

[203]  Guang-Zhong Yang,et al.  Real-time intra-operative 3D tissue deformation recovery , 2008, 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[204]  U. Mezger,et al.  Navigation in surgery , 2013, Langenbeck's Archives of Surgery.

[205]  C Wu,et al.  A full geometric and photometric calibration method for oblique-viewing endoscopes , 2010, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[206]  Shang-Hong Lai,et al.  Automatic Distortion Correction of Endoscopic Images Captured With Wide-Angle Zoom Lens , 2013, IEEE Transactions on Biomedical Engineering.

[207]  Guang-Zhong Yang,et al.  Practical Intraoperative Stereo Camera Calibration , 2014, MICCAI.

[208]  Cüneyt Güzelis,et al.  Semiautomatic Transfer Function Initialization for Abdominal Visualization Using Self-Generating Hierarchical Radial Basis Function Networks , 2009, IEEE Transactions on Visualization and Computer Graphics.

[209]  D. Louis Collins,et al.  The state of the art of visualization in mixed reality image guided surgery , 2013, Comput. Medical Imaging Graph..

[210]  Guang-Zhong Yang,et al.  Augmented Reality Image Guidance in Minimally Invasive Prostatectomy , 2010, Prostate Cancer Imaging.

[211]  Guang-Zhong Yang,et al.  pq-space Based Non-Photorealistic Rendering for Augmented Reality , 2007, MICCAI.

[212]  Rutger M. Schols,et al.  Advanced intraoperative imaging methods for laparoscopic anatomy navigation: an overview , 2012, Surgical Endoscopy.

[213]  Adrien Bartoli,et al.  Towards Live Monocular 3D Laparoscopy Using Shading and Specularity Information , 2012, IPCAI.

[214]  Michael Figl,et al.  Reconstruction of a 3D surface from video that is robust to missing data and outliers: Application to minimally invasive surgery using stereo and mono endoscopes , 2012, Medical Image Anal..

[215]  Michael Figl,et al.  Non-rigid Reconstruction of the Beating Heart Surface for Minimally Invasive Cardiac Surgery , 2009, MICCAI.

[216]  Anders Ynnerman,et al.  Uncertainty Visualization in Medical Volume Rendering Using Probabilistic Animation , 2007, IEEE Transactions on Visualization and Computer Graphics.

[217]  Jacques Marescaux,et al.  Transatlantic robot-assisted telesurgery , 2001, Nature.

[218]  Philippe Cinquin,et al.  Multi-view vision system for laparoscopy surgery , 2015, International Journal of Computer Assisted Radiology and Surgery.

[219]  Makoto Hashizume,et al.  Realtime Organ Tracking for Endoscopic Augmented Reality Visualization Using Miniature Wireless Magnetic Tracker , 2008, MIAR.

[220]  Kwan-Liu Ma,et al.  Visibility Histograms and Visibility-Driven Transfer Functions , 2011, IEEE Transactions on Visualization and Computer Graphics.

[221]  Robert Rohling,et al.  Motion of the Kidney Between Preoperative and Intraoperative Positioning , 2013, IEEE Transactions on Biomedical Engineering.

[222]  Nassir Navab,et al.  Improving Depth Perception in Medical AR , 2007, Bildverarbeitung für die Medizin.

[223]  Sukryool Kang,et al.  Evaluation of Electromagnetic Tracking for Stereoscopic Augmented Reality Laparoscopic Visualization , 2014, CLIP@MICCAI.

[224]  Michael Figl,et al.  Image guidance for robotic minimally invasive coronary artery bypass , 2010, Comput. Medical Imaging Graph..

[225]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[226]  Toril A. Nagelhus.  Laparoscopic ultrasound: a survey of its current and future use, requirements, and integration with navigation technology , 2010 .

[227]  Luc Soler,et al.  A structured light system to guide percutaneous punctures in interventional radiology , 2008, SPIE Photonics Europe.

[228]  Nazim Haouchine,et al.  Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[229]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[230]  Lena Maier-Hein,et al.  Pose-independent surface matching for intra-operative soft-tissue marker-less registration , 2014, Medical Image Anal..

[231]  Ken Masamune,et al.  An effective visualization technique for depth perception in augmented reality‐based surgical navigation , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[232]  Robert Rohling,et al.  Calibration and Stereo Tracking of a Laparoscopic Ultrasound Transducer for Augmented Reality in Surgery , 2013, AE-CAI.

[233]  Lena Maier-Hein,et al.  Physics-based shape matching for intraoperative image guidance. , 2014, Medical physics.

[234]  Daniel Mirota,et al.  Evaluation of a System for High-Accuracy 3D Image-Based Registration of Endoscopic Video to C-Arm Cone-Beam CT for Image-Guided Skull Base Surgery , 2013, IEEE Transactions on Medical Imaging.

[235]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[236]  Lena Maier-Hein,et al.  Robust augmented reality guidance with fluorescent markers in laparoscopic surgery , 2016, International Journal of Computer Assisted Radiology and Surgery.

[237]  Lena Maier-Hein,et al.  Time-of-flight camera technique for augmented reality in computer-assisted interventions , 2011, Medical Imaging.

[238]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[239]  David A Jaffray,et al.  Improving image-guided target localization through deformable registration , 2008, Acta oncologica.

[240]  Stefan Weber,et al.  A clinically applicable laser-based image-guided system for laparoscopic liver procedures , 2016, International Journal of Computer Assisted Radiology and Surgery.

[241]  Thomas Lange,et al.  3D ultrasound-CT registration of the liver using combined landmark-intensity information , 2008, International Journal of Computer Assisted Radiology and Surgery.

[242]  Philippe Poignet,et al.  Three-dimensional heart motion estimation using endoscopic monocular vision system: From artificial landmarks to texture analysis , 2007, Biomed. Signal Process. Control..

[243]  Yuncheng You,et al.  Video‐based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[244]  Mamoru Mitsuishi,et al.  3D Shape Reconstruction Endoscope using Shape from Focus , 2009, VISAPP.

[245]  Guang-Zhong Yang,et al.  Intra-Operative Visualizations: Perceptual Fidelity and Human Factors , 2008, Journal of Display Technology.

[246]  Sebastian Bodenstedt,et al.  Image-based laparoscopic bowel measurement , 2016, International Journal of Computer Assisted Radiology and Surgery.

[247]  Jae Young Lee,et al.  Automatic registration between 3D intra-operative ultrasound and pre-operative CT images of the liver based on robust edge matching , 2012, Physics in medicine and biology.

[248]  Makoto Hashizume,et al.  Augmented reality navigation system for laparoscopic splenectomy in children based on preoperative CT image using optical tracking device , 2012, Pediatric Surgery International.

[249]  Daniel S. Elson,et al.  An endoscopic structured light system using multispectral detection , 2015, International Journal of Computer Assisted Radiology and Surgery.

[250]  Sebastian Bodenstedt,et al.  Superpixel-based structure classification for laparoscopic surgery , 2016, SPIE Medical Imaging.

[251]  Ole Vegard Solberg,et al.  Navigated laparoscopic ultrasound in abdominal soft tissue surgery: technological overview and perspectives , 2012, International Journal of Computer Assisted Radiology and Surgery.

[252]  Sébastien Ourselin,et al.  Accuracy validation of an image guided laparoscopy system for liver resection , 2015, Medical Imaging.

[253]  Beat P. Müller-Stich,et al.  Computer-assisted abdominal surgery: new technologies , 2015, Langenbeck's Archives of Surgery.

[254]  Rudy J Lapeer,et al.  Using a passive coordinate measurement arm for motion tracking of a rigid endoscope for augmented‐reality image‐guided surgery , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[255]  Makoto Hashizume,et al.  A real-time navigation system for laparoscopic surgery based on three-dimensional ultrasound using magneto-optic hybrid tracking configuration , 2007, International Journal of Computer Assisted Radiology and Surgery.

[256]  Masashi Baba,et al.  2-DOF auto-calibration for a 3D endoscope system based on active stereo , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[257]  Guang-Zhong Yang,et al.  Visual Force Feedback for Hand-Held Microsurgical Instruments , 2015, MICCAI.

[258]  Yu Sun,et al.  Unobtrusive Augmentation of Critical Hidden Structures in Laparoscopy , 2014, MMVR.

[259]  Wolfgang Birkfellner,et al.  Electromagnetic Tracking in Medicine—A Review of Technology, Validation, and Applications , 2014, IEEE Transactions on Medical Imaging.

[260]  H. Bülthoff,et al.  3D shape perception from combined depth cues in human visual cortex , 2005, Nature Neuroscience.

[261]  Nassir Navab,et al.  Auditory and Visio-Temporal Distance Coding for 3-Dimensional Perception in Medical Augmented Reality , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[262]  Robert L Galloway,et al.  Toward image guided robotic surgery: system validation. , 2009, The Journal of urology.

[263]  Kai-Che Liu,et al.  Real-time advanced spinal surgery via visible patient model and augmented reality system , 2014, Comput. Methods Programs Biomed..

[264]  Michael J. Daly,et al.  Surgeons blinded by enhanced navigation: the effect of augmented reality on attention , 2013, Surgical Endoscopy.

[265]  Daniel Mirota,et al.  An on-board surgical tracking and video augmentation system for C-arm image guidance , 2012, International Journal of Computer Assisted Radiology and Surgery.

[266]  D. Rueckert,et al.  Medical Image Registration , 2010 .

[267]  João Pedro Barreto,et al.  Real-time HD image distortion correction in heterogeneous parallel computing systems using efficient memory access patterns , 2012, Journal of Real-Time Image Processing.

[268]  Luc Soler,et al.  An Augmented Reality Framework for Soft Tissue Surgery , 2014, MICCAI.

[269]  Nazim Haouchine,et al.  Framework for augmented reality in Minimally Invasive laparoscopic surgery , 2015, 2015 17th International Conference on E-health Networking, Application & Services (HealthCom).

[270]  Lena Maier-Hein,et al.  Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[271]  Yoshinobu Sato,et al.  Augmented reality navigation system for endoscopic surgery based on three-dimensional ultrasound and computed tomography: Application to 20 clinical cases , 2005 .

[272]  Terry M. Peters,et al.  Robust Intraoperative US Probe Tracking Using a Monocular Endoscopic Camera , 2013, MICCAI.

[273]  Raúl San José Estépar,et al.  Real‐time computed tomography‐based augmented reality for natural orifice transluminal endoscopic surgery navigation , 2012, The British journal of surgery.

[274]  Sébastien Ourselin,et al.  Locally rigid, vessel-based registration for laparoscopic liver surgery , 2015, International Journal of Computer Assisted Radiology and Surgery.

[275]  Heinz-Otto Peitgen,et al.  Assessment of Intraoperative Liver Deformation During Hepatic Resection: Prospective Clinical Study , 2010, World Journal of Surgery.