The Future of Endoscopic Navigation: A Review of Advanced Endoscopic Vision Technology

Minimally invasive medicine has become mainstream because of its crucial clinical significance in providing a low risk of postoperative complications, limited blood loss, short postoperative recovery time, and small sizes of associated physiological tissue wounds. Endoscopic navigation systems comprise a research hot spot in medical science and technology and are an essential means to achieve precision medicine and improve surgical operation safety. As a core component in endoscopic navigation during minimally invasive surgery, endoscopes play a critical role in disease diagnosis and treatment. The development of endoscopic vision technologies has resulted in a renewed drive to further develop endoscopic navigation systems. Multiple endoscopic optical imaging modalities provide data sources for endoscopic vision technology, including white-light endoscopy, contrast-enhanced imaging and technologies involving magnified observation. Endoscopic vision is a specific application of computer vision involving the use of endoscopes that include instrument tracking, endoscopic view expansion, and suspicious lesion tracking in the application of endoscopic navigation. These techniques help surgeons or surgical robots locate instruments and lesions and expand the field of view of the endoscope. Although these technologies have been applied to various clinical and pre-clinical diagnoses and treatments, the use and combination of these advanced technologies in endoscopic navigation system for specific clinical requirements remains challenging. This review performs a broad survey of advanced endoscopic vision technologies and their application in endoscopic navigation systems. Finally, we discuss the challenges and future directions in implementing and developing endoscopic navigation systems.

[1]  Paul R. Cohen,et al.  Camera Calibration with Distortion Models and Accuracy Evaluation , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[2]  Yuwei Wang,et al.  Camera Calibration Robust to Defocus Using Phase-Shifting Patterns , 2017, Sensors.

[3]  Seyed Shahim Vedaei,et al.  A Fluorescence-Based Wireless Capsule Endoscopy System for Detecting Colorectal Cancer , 2020, Cancers.

[4]  Rob H N Tijssen,et al.  Particle Filter-Based Target Tracking Algorithm for Magnetic Resonance-Guided Respiratory Compensation: Robustness and Accuracy Assessment. , 2018, International journal of radiation oncology, biology, physics.

[5]  Zijian Zhao,et al.  Image-based laparoscopic tool detection and tracking using convolutional neural networks: a review of the literature , 2020, Computer assisted surgery.

[6]  Marcin Markuszewski,et al.  Usage of invisible near infrared light (NIR) fluorescence with indocyanine green (ICG) and methylene blue (MB) in urological oncology. Part 1 , 2014, Central European journal of urology.

[7]  Luc Soler,et al.  Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing , 2003, IEEE Trans. Robotics Autom..

[8]  J. Marlow.  History of laparoscopy, optics, fiberoptics, and instrumentation. , 1976, Clinical obstetrics and gynecology.

[9]  T. Robinson,et al.  Minimally invasive surgery , 1999, European Surgical Research.

[10]  Koichi Yamazaki,et al.  CT-guided transbronchial biopsy using an ultrathin bronchoscope with virtual bronchoscopic navigation. , 2004, Chest.

[11]  Bin Sun,et al.  Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy , 2018, Endoscopy.

[12]  Nazim Haouchine,et al.  Towards an accurate tracking of liver tumors for augmented reality in robotic assisted surgery , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[13]  Alexandre Hostettler,et al.  SLAM based Quasi Dense Reconstruction For Minimally Invasive Surgery Scenes , 2017, ArXiv.

[14]  Danail Stoyanov,et al.  Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers , 2019, IEEE Robotics and Automation Letters.

[15]  K. Takano,et al.  Endoscopy-assisted transoral resection of a parapharyngeal space schwannoma without mandibular dissection. , 2020, Auris, nasus, larynx.

[16]  Nobuhiko Hata,et al.  Computer vision-guided bronchoscopic navigation using dual CNN-generated depth images and ICP registration , 2020, Medical Imaging: Image-Guided Procedures.

[17]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Vincent Lepetit,et al.  Predicting Sharp and Accurate Occlusion Boundaries in Monocular Depth Estimation Using Displacement Fields , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Nick Barnes,et al.  Unsupervised Learning of Endoscopy Video Frames' Correspondences from Global and Local Transformation , 2018, OR 2.0/CARE/CLIP/ISIC@MICCAI.

[20]  Takeo Kanade,et al.  Accurate camera calibration using iterative refinement of control points , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[21]  Huilong Duan,et al.  Global and Local Panoramic Views for Gastroscopy: An Assisted Method of Gastroscopic Lesion Surveillance , 2015, IEEE Transactions on Biomedical Engineering.

[22]  Fei Yan,et al.  An improved robust hand-eye calibration for endoscopy navigation system , 2016, SPIE Medical Imaging.

[23]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  J. M. M. Montiel,et al.  ORB-SLAM: A Versatile and Accurate Monocular SLAM System , 2015, IEEE Transactions on Robotics.

[25]  Makoto Watanabe,et al.  Laparoscopic Liver Surgery Guided by Virtual Real-time CT-Guided Volume Navigation , 2020, Journal of Gastrointestinal Surgery.

[26]  K. Ohtsuka,et al.  Development and Validation of a Deep Neural Network for Accurate Evaluation of Endoscopic Images From Patients with Ulcerative Colitis. , 2020, Gastroenterology.

[27]  Peter M. Full,et al.  Robust Medical Instrument Segmentation Challenge 2019 , 2020, ArXiv.

[28]  Danail Stoyanov,et al.  Vision‐based and marker‐less surgical tool detection and tracking: a review of the literature , 2017, Medical Image Anal..

[29]  Bradley J. Nelson,et al.  Visual-Kinematic Monocular SLAM using a Magnetic Endoscope , 2018 .

[30]  S. Qazi,et al.  Image Guided Endoscopic Sinus Surgery: First Experience from Kashmir Valley , 2020, Indian Journal of Otolaryngology and Head & Neck Surgery.

[31]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Henry Horng-Shing Lu,et al.  Accurate Classification of Diminutive Colorectal Polyps Using Computer-Aided Analysis. , 2017, Gastroenterology.

[33]  Paul Fockens,et al.  Endoscopic video autofluorescence imaging may improve the detection of early neoplasia in patients with Barrett's esophagus. , 2005, Gastrointestinal endoscopy.

[34]  Hongliang Ren,et al.  Endoscope Navigation and 3D Reconstruction of Oral Cavity by Visual SLAM with Mitigated Data Scarcity , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[35]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[36]  Ève Coste-Manière,et al.  Towards endoscopic augmented reality for robotically assisted minimally invasive cardiac surgery , 2001, Proceedings International Workshop on Medical Imaging and Augmented Reality.

[37]  Arie E. Kaufman,et al.  Depth Reconstruction and Computer-Aided Polyp Detection in Optical Colonoscopy Video Frames , 2016, ArXiv.

[38]  H. Ashrafian,et al.  A systematic review of left ventricular cardio-endoscopic surgery , 2017, Journal of Cardiothoracic Surgery.

[39]  Huilong Duan,et al.  Real-time gastric polyp detection using convolutional neural networks , 2019, PloS one.

[40]  A. Saftoiu,et al.  Computer Aided Diagnosis for Confocal Laser Endomicroscopy in Advanced Colorectal Adenocarcinoma , 2016, PloS one.

[41]  Harley Chan,et al.  Real‐time tracking and virtual endoscopy in cone‐beam CT‐guided surgery of the sinuses and skull base in a cadaver model , 2011, International forum of allergy & rhinology.

[42]  Yue Yang,et al.  Using distortion correction to improve the precision of camera calibration , 2019 .

[43]  K. Mori,et al.  Simultaneous detection and characterization of diminutive polyps with the use of artificial intelligence during colonoscopy , 2019, VideoGIE : an official video journal of the American Society for Gastrointestinal Endoscopy.

[44]  Takumi Itoh,et al.  Deep learning analyzes Helicobacter pylori infection by upper gastrointestinal endoscopy images , 2018, Endoscopy International Open.

[45]  Gustavo Carneiro,et al.  Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue , 2016, ECCV.

[46]  K. Mori,et al.  Current status and future perspective on artificial intelligence for lower endoscopy , 2020, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[47]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[48]  Bing Hu,et al.  Real-time automated diagnosis of precancerous lesion and early esophageal squamous cell carcinoma using a deep learning model (with videos). , 2019, Gastrointestinal endoscopy.

[49]  R. Bisschops,et al.  Leaving colorectal polyps in place can be achieved with high accuracy using blue light imaging (BLI) , 2018, United European gastroenterology journal.

[50]  Olivier Stasse,et al.  MonoSLAM: Real-Time Single Camera SLAM , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Florent Nageotte,et al.  Segmentation and Guidance of Multiple Rigid Objects for Intra-operative Endoscopic Vision , 2006, WDV.

[52]  Huilong Duan,et al.  Surface Reconstruction from Tracked Endoscopic Video Using the Structure from Motion Approach , 2013, AE-CAI.

[53]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[54]  Nikolaos Doulamis,et al.  Deep Learning for Computer Vision: A Brief Review , 2018, Comput. Intell. Neurosci..

[55]  H. Kawachi,et al.  Artificial intelligence diagnosis of Helicobacter pylori infection using blue laser imaging-bright and linked color imaging: a single-center prospective study , 2018, Annals of gastroenterology.

[56]  Masatoshi Okutomi,et al.  3D Reconstruction of Whole Stomach from Endoscope Video Using Structure-from-Motion , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[57]  Jean-Nicolas Ouellet,et al.  Precise ellipse estimation without contour point extraction , 2009, Machine Vision and Applications.

[58]  Yaser Sheikh,et al.  Deltille Grids for Geometric Camera Calibration , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[59]  Russell H. Taylor,et al.  Endoscopic navigation in the absence of CT imaging , 2018, MICCAI.

[60]  Dean Brown,et al.  Decentering distortion of lenses , 1966 .

[61]  Pascal Fua,et al.  Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery , 2017, MICCAI.

[62]  Abed Malti,et al.  Robust hand-eye calibration for computer aided medical endoscopy , 2010, 2010 IEEE International Conference on Robotics and Automation.

[63]  Janne Heikkilä,et al.  Geometric Camera Calibration Using Circular Control Points , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[64]  Xi Zheng,et al.  Unsupervised-Learning-Based Continuous Depth and Motion Estimation With Monocular Endoscopy for Virtual Reality Minimally Invasive Surgery , 2020, IEEE Transactions on Industrial Informatics.

[65]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[66]  Monan Wang,et al.  A review of computer‐assisted orthopaedic surgery systems , 2020, The international journal of medical robotics + computer assisted surgery : MRCAS.

[67]  Jason J. Corso,et al.  Product of tracking experts for visual tracking of surgical tools , 2013, 2013 IEEE International Conference on Automation Science and Engineering (CASE).

[68]  T. Gildea,et al.  Navigational bronchoscopy: a guide through history, current use, and developing technology , 2020, Journal of thoracic disease.

[69]  Ralf Kiesslich,et al.  Advances in endoscopic imaging of colorectal neoplasia. , 2010, Gastroenterology.

[70]  Clyde Matava,et al.  A Convolutional Neural Network for Real Time Classification, Identification, and Labelling of Vocal Cord and Tracheal Using Laryngoscopy and Bronchoscopy Video , 2020, Journal of Medical Systems.

[71]  Oleg Sergiyenko,et al.  Improve three-dimensional point localization accuracy in stereo vision systems using a novel camera calibration method , 2020 .

[72]  Alexandre Hostettler,et al.  ORBSLAM-Based Endoscope Tracking and 3D Reconstruction , 2016, CARE@MICCAI.

[73]  Jonathan C Irish,et al.  Novel Thoracoscopic Navigation System With Augmented Real-Time Image Guidance for Chest Wall Tumors. , 2018, The Annals of thoracic surgery.

[74]  Andreas Geiger,et al.  Automatic camera and range sensor calibration using a single shot , 2012, 2012 IEEE International Conference on Robotics and Automation.

[75]  Rongguang Liang,et al.  Optical Design for Biomedical Imaging , 2011 .

[76]  Christoph Schmalz,et al.  Camera calibration: active versus passive targets , 2011 .

[77]  S. Dawsey,et al.  A tablet-interfaced high-resolution microendoscope with automated image interpretation for real-time evaluation of esophageal squamous cell neoplasia. , 2016, Gastrointestinal endoscopy.

[78]  K. Mori,et al.  Fully automated diagnostic system with artificial intelligence using endocytoscopy to identify the presence of histologic inflammation associated with ulcerative colitis (with video). , 2019, Gastrointestinal endoscopy.

[79]  S. Hirota,et al.  Ex vivo comparison of angioscopy and histopathology for the evaluation of coronary plaque characteristics , 2016, The International Journal of Cardiovascular Imaging.

[80]  J. M. M. Montiel,et al.  Visual SLAM for Handheld Monocular Endoscope , 2014, IEEE Transactions on Medical Imaging.

[81]  B. Hayee,et al.  The future of endoscopy: Advances in endoscopic image innovations , 2020, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[82]  J. Akulian,et al.  Peripheral Lung Nodule Diagnosis and Fiducial Marker Placement Using a Novel Tip-Tracked Electromagnetic Navigation Bronchoscopy System , 2019, Journal of bronchology & interventional pulmonology.

[83]  Ronan A. Cahill,et al.  Near-infrared (NIR) perfusion angiography in minimally invasive colorectal surgery , 2014, Surgical Endoscopy.

[84]  Sébastien Ourselin,et al.  Toward Detection and Localization of Instruments in Minimally Invasive Surgery , 2013, IEEE Transactions on Biomedical Engineering.

[85]  J. Bladowska,et al.  Navigated neuroendoscopy combined with intraoperative magnetic resonance cysternography for treatment of arachnoid cysts , 2019, Neurosurgical review.

[86]  Jaesoon Choi,et al.  Robust Real-Time Detection of Laparoscopic Instruments in Robot Surgery Using Convolutional Neural Networks with Motion Vector Prediction , 2019, Applied Sciences.

[87]  T. Berzin,et al.  Real-time automatic detection system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study , 2019, Gut.

[88]  Kensaku Mori,et al.  Advanced Endoscopic Navigation: Surgical Big Data, Methodology, and Applications. , 2018, Annual review of biomedical engineering.

[89]  Meic H. Schmidt,et al.  Placement of Ommaya Reservoirs Using Electromagnetic Neuronavigation and Neuroendoscopy: A Retrospective Study with Cost-Benefit Analysis. , 2019, World neurosurgery.

[90]  Jie Yang,et al.  Stereo Endoscopic Image Super-Resolution Using Disparity-Constrained Parallel Attention , 2020, ArXiv.

[91]  Masatoshi Okutomi,et al.  Stomach 3D Reconstruction Based on Virtual Chromoendoscopic Image Generation , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[92]  Oisin Mac Aodha,et al.  Unsupervised Monocular Depth Estimation with Left-Right Consistency , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[93]  Eun Mi Song,et al.  Endoscopic diagnosis and treatment planning for colorectal polyps using a deep-learning model , 2020, Scientific Reports.

[94]  Sven Haase,et al.  Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[95]  Jenny Dankelman,et al.  In-vivo real-time tracking of surgical instruments in endoscopic video , 2012, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[96]  Helder Araujo,et al.  EndoSLAM Dataset and An Unsupervised Monocular Visual Odometry and Depth Estimation Approach for Endoscopic Videos: Endo-SfMLearner , 2020 .

[97]  Helder Araújo,et al.  A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots , 2017, International Journal of Intelligent Robotics and Applications.

[98]  David Zhang,et al.  Fast Visual Tracking via Dense Spatio-temporal Context Learning , 2014, ECCV.

[99]  Xiongxiong He,et al.  WCE polyp detection based on novel feature descriptor with normalized variance locality-constrained linear coding , 2020, International Journal of Computer Assisted Radiology and Surgery.

[100]  Joost van der Putten,et al.  Deep Learning Biopsy Marking of Early Neoplasia in Barrett’s Esophagus by Combining WLE and BLI Modalities , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[101]  KANAKO HARADA,et al.  ICG fluorescence endoscope for visualization of the placental vascular network , 2009, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[102]  M. Fujishiro,et al.  Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images , 2018, Gastric Cancer.

[103]  Mohammad Hossein Khosravi,et al.  Global, regional, and national age-sex-specific mortality for 282 causes of death in 195 countries and territories, 1980–2017: a systematic analysis for the Global Burden of Disease Study 2017 , 2018, Lancet.

[104]  Roberto A. Hexsel,et al.  Particle filtering in the Hough space for instrument tracking , 2012, Comput. Biol. Medicine.

[105]  Guang-Zhong Yang,et al.  Real-Time Stereo Reconstruction in Robotically Assisted Minimally Invasive Surgery , 2010, MICCAI.

[106]  Janne Heikkilä,et al.  A four-step camera calibration procedure with implicit image correction , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[107]  Carsten Rother,et al.  PatchMatch Stereo - Stereo Matching with Slanted Support Windows , 2011, BMVC.

[108]  E. J. Aslim,et al.  The Utility of Intraoperative Near Infrared Fluorescence (NIR) Imaging with Indocyanine Green (ICG) for the Assessment of Kidney Allograft Perfusion , 2018, Journal of transplantation.

[109]  Michael B. Wallace,et al.  High-definition and high-magnification endoscopes. , 2014, Gastrointestinal endoscopy.

[110]  T. Berzin,et al.  Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy , 2018, Nature Biomedical Engineering.

[111]  Y. Uchida.  Recent advances in coronary angioscopy. , 2011, Journal of cardiology.

[112]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[113]  D. Hogarth.  Use of augmented fluoroscopic imaging during diagnostic bronchoscopy. , 2018, Future oncology.

[114]  J. Rey,et al.  Narrow-Band Imaging: Potential and Limitations , 2006, Endoscopy.

[115]  Wei Cai,et al.  Augmented reality navigation for liver resection with a stereoscopic laparoscope , 2020, Comput. Methods Programs Biomed..

[116]  R. Weersink,et al.  Localization of pulmonary nodules using navigation bronchoscope and a near-infrared fluorescence thoracoscope. , 2015, Annals of Thoracic Surgery.

[117]  Wei Liu,et al.  Comparison in efficacy and safety of forceps biopsy for peripheral lung lesions guided by endobronchial ultrasound-guided sheath (EBUS-GS) and electromagnetic navigation bronchoscopy combined with EBUS (ENB-EBUS). , 2020, American journal of translational research.

[118]  S. Kudo,et al.  Endocytoscopy , 2020, Endoscopic Management of Colorectal T1(SM) Carcinoma.

[119]  Yuan-Fang Wang,et al.  Image analysis for automated tracking in robot-assisted endoscopic surgery , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[120]  Jing Cheng,et al.  Application of convolutional neural network in the diagnosis of the invasion depth of gastric cancer based on conventional endoscopy. , 2019, Gastrointestinal endoscopy.

[121]  William E. Higgins,et al.  Autofluorescence Bronchoscopy Video Analysis for Lesion Frame Detection* , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[122]  M. Fujishiro,et al.  Diagnosis using deep-learning artificial intelligence based on the endocytoscopic observation of the esophagus , 2018, Esophagus.

[123]  J. Borovička,et al.  Autofluorescence endoscopy in surveillance of Barrett's esophagus: a multicenter randomized trial on diagnostic efficacy. , 2006, Endoscopy.

[124]  Richard J. Chen,et al.  SLAM Endoscopy enhanced by adversarial depth prediction , 2019, ArXiv.

[125]  G. Falk,et al.  Autofluorescence endoscopy. , 2009, Gastrointestinal endoscopy clinics of North America.

[126]  Russell H. Taylor,et al.  Self-supervised Learning for Dense Depth Estimation in Monocular Endoscopy , 2018, OR 2.0/CARE/CLIP/ISIC@MICCAI.

[127]  Next generation of endoscopy: Harmony with artificial intelligence and robotic‐assisted devices , 2020, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[128]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[129]  G. Hirzinger,et al.  Real-time visual servoing for laparoscopic surgery. Controlling robot motion with color image segmentation , 1997, IEEE Engineering in Medicine and Biology Magazine.

[130]  Roland Siegwart,et al.  Automatic detection of checkerboards on blurred and distorted images , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[131]  B. Thompson,et al.  Multimodal laser-based angioscopy for structural, chemical and biological imaging of atherosclerosis , 2017, Nature Biomedical Engineering.

[132]  Mark Rentschler,et al.  Autonomous Localization, Navigation and Haustral Fold Detection for Robotic Endoscopy , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[133]  Noah Snavely,et al.  Unsupervised Learning of Depth and Ego-Motion from Video , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[134]  Face and content validity of a novel, web-based otoscopy simulator for medical education , 2015, Journal of Otolaryngology - Head & Neck Surgery.

[135]  F. Jia,et al.  Real-time navigation for laparoscopic hepatectomy using image fusion of preoperative 3D surgical plan and intraoperative indocyanine green fluorescence imaging , 2019, Surgical Endoscopy.

[136]  E. Schoon,et al.  Improved Barrett's neoplasia detection using computer-assisted multiframe analysis of volumetric laser endomicroscopy. , 2019, Diseases of the esophagus : official journal of the International Society for Diseases of the Esophagus.

[137]  Junzhou Huang,et al.  Deep learning based multi-label classification for surgical tool presence detection in laparoscopic videos , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[138]  Rafeef Abugharbieh,et al.  Robust Dense Endoscopic Stereo Reconstruction for Minimally Invasive Surgery , 2012, MCV.

[139]  Yanli Wang,et al.  Percutaneous nephrolithotripsy: C-arm CT with 3D virtual navigation in non-dilated renal collecting systems. , 2018, Diagnostic and interventional radiology.

[140]  Y. Zhong,et al.  Using deep learning system in endoscopy for screening of early esophageal squamous cell carcinoma (with video). , 2019, Gastrointestinal endoscopy.

[141]  Abbas F. Sadikot,et al.  IBIS: an OR ready open-source platform for image-guided neurosurgery , 2017, International Journal of Computer Assisted Radiology and Surgery.

[142]  Alexandre Hostettler,et al.  Live Tracking and Dense Reconstruction for Handheld Monocular Endoscopy , 2019, IEEE Transactions on Medical Imaging.

[143]  Jan Tesarik,et al.  The value of virtual reality simulators in hysteroscopy and training capacity: a systematic review , 2020, Minimally invasive therapy & allied technologies : MITAT : official journal of the Society for Minimally Invasive Therapy.

[144]  Zhanyu Ma,et al.  Computer-aided diagnosis of colorectal polyps using linked color imaging colonoscopy to predict histology , 2019, Scientific Reports.

[145]  Long Chen,et al.  Real-time geometry-aware augmented reality in minimally invasive surgery , 2017, Healthcare technology letters.

[146]  Haoyin Zhou,et al.  Real-Time Dense Reconstruction of Tissue Surface From Stereo Optical Video , 2020, IEEE Transactions on Medical Imaging.

[147]  Bin Chen,et al.  Camera calibration using synthetic random speckle pattern and digital image correlation , 2020 .

[148]  S. Weber,et al.  Efficiency, Accuracy and Clinical Applicability of a New Image-Guided Surgery System in 3D Laparoscopic Liver Surgery , 2019, Journal of Gastrointestinal Surgery.

[149]  Danail Stoyanov,et al.  Adjoint Transformation Algorithm for Hand–Eye Calibration with Applications in Robotic Assisted Surgery , 2018, Annals of Biomedical Engineering.

[150]  S. Jacques,et al.  Optical reflectance and transmittance of tissues: principles and applications , 1990 .

[151]  Sébastien Ourselin,et al.  Hand–eye calibration for rigid laparoscopes using an invariant point , 2016, International Journal of Computer Assisted Radiology and Surgery.

[152]  M. Monici.  Cell and tissue autofluorescence research and diagnostic applications. , 2005, Biotechnology annual review.

[153]  Hyunseok Choi,et al.  Comparative study of hand–eye calibration methods for augmented reality using an endoscope , 2018, J. Electronic Imaging.

[154]  Toshimitsu Kaneko,et al.  Deep monocular 3D reconstruction for assisted navigation in bronchoscopy , 2017, International Journal of Computer Assisted Radiology and Surgery.

[155]  Stefan Weber,et al.  Laparoscopic image-based navigation for microwave ablation of liver tumors—A multi-center study , 2017, Surgical Endoscopy.

[156]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[157]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[158]  Yu Sun,et al.  Simultaneous Tracking, 3D Reconstruction and Deforming Point Detection for Stereoscope Guided Surgery , 2013, AE-CAI.

[159]  Jingang Zhang,et al.  Deep Convolutional Network for Stereo Depth Mapping in Binocular Endoscopy , 2020, IEEE Access.

[160]  Qinyong Lin,et al.  [Automatic Robotic Puncture System for Accurate Liver Cancer Ablation Based on Optical Surgical Navigation]. , 2018, Zhongguo yi liao qi xie za zhi = Chinese journal of medical instrumentation.

[161]  Pinar Civicioglu,et al.  Camera calibration by using weighted differential evolution algorithm: a comparative study with ABC, PSO, COBIDE, DE, CS, GWO, TLBO, MVMO, FOA, LSHADE, ZHANG and BOUGUET , 2020, Neural Computing and Applications.

[162]  Danail Stoyanov,et al.  Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy , 2019, International Journal of Computer Assisted Radiology and Surgery.

[163]  Vivek Kaul,et al.  Confocal laser endomicroscopy. , 2009, Gastrointestinal endoscopy.

[164]  Sandrine Voros,et al.  Surgical tool tracking based on two CNNs: from coarse to fine , 2019 .

[165]  Ryan J. Halter,et al.  Multi-modal Framework for Image-guided Trans-oral Surgery with Intraoperative Imaging and Deformation Modeling* , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[166]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[167]  Kazuki Sumiyama,et al.  Real-time computer-aided diagnosis of diminutive rectosigmoid polyps using an auto-fluorescence imaging system and novel color intensity analysis software , 2019, Scandinavian journal of gastroenterology.

[168]  Liping Yao,et al.  Application of cardiovascular virtual endoscopy: a pilot study on roaming path planning for diagnosis of congenital heart diseases in children , 2018, Scientific Reports.

[169]  Peter H. N. de With,et al.  Hand-Eye Camera Calibration with an Optical Tracking System , 2018, ICDSC.

[170]  Helder Araújo,et al.  Magnetic- Visual Sensor Fusion-based Dense 3D Reconstruction and Localization for Endoscopic Capsule Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[171]  Aly A. Farag,et al.  CSIFT: A SIFT Descriptor with Color Invariant Characteristics , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[172]  Sungwan Kim,et al.  Development of stereo endoscope system with its innovative master interface for continuous surgical operation , 2017, Biomedical engineering online.

[173]  Qican Zhang,et al.  Camera calibration with active phase target: improvement on feature detection and optimization. , 2013, Optics letters.

[174]  K. Chayama,et al.  Quantitative identification of mucosal gastric cancer under magnifying endoscopy with flexible spectral imaging color enhancement , 2013, Journal of gastroenterology and hepatology.

[175]  Russell H. Taylor,et al.  Endoscopic-CT: learning-based photometric reconstruction for endoscopic sinus surgery , 2016, SPIE Medical Imaging.

[176]  Peng Wang,et al.  Novel endoscopic optical diagnostic technologies in medical trial research: recent advancements and future prospects , 2021, Biomedical engineering online.

[177]  Peter H. N. de With,et al.  Image fusion on the endoscopic view for endo-nasal skull-base surgery , 2019, Medical Imaging.

[178]  Faisal Mahmood,et al.  Deep learning-based depth estimation from a synthetic endoscopy image training set , 2018, Medical Imaging.

[179]  Dan Stoianovici,et al.  Endoscopic Stone Measurement During Ureteroscopy. , 2017, Journal of endourology.

[180]  Srikumar Ramalingam,et al.  A Unifying Model for Camera Calibration , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[181]  Ichiro Sakuma,et al.  Vision-based endoscope tracking for 3D ultrasound image-guided surgical navigation , 2015, Comput. Medical Imaging Graph..

[182]  U. Thomale,et al.  Augmented reality in intraventricular neuroendoscopy , 2017, Acta Neurochirurgica.

[183]  Aimin Li,et al.  Artificial Intelligence-Assisted Colonoscopy for Detection of Colon Polyps: a Prospective, Randomized Cohort Study , 2020, Journal of Gastrointestinal Surgery.

[184]  Terry M. Peters,et al.  Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: creating stereoscopic panoramas , 2002, IEEE Transactions on Medical Imaging.

[185]  N. Kato,et al.  A computer-aided diagnosis system using only white-light endoscopy for the prediction of invasion depth in colorectal cancer. , 2020, Gastrointestinal endoscopy.

[186]  Quanxin Liu,et al.  Camera calibration method based on optimal polarization angle , 2019, Optics and Lasers in Engineering.

[187]  M. Fujishiro,et al.  Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks. , 2019, Gastrointestinal endoscopy.

[188]  Cheng Wang,et al.  Visual SLAM for bronchoscope tracking and bronchus reconstruction in bronchoscopic navigation , 2019, Medical Imaging.

[189]  Guoyan Zheng,et al.  Landmark‐based augmented reality system for paranasal and transnasal endoscopic surgeries , 2009, The international journal of medical robotics + computer assisted surgery : MRCAS.

[190]  Yuelong Liang,et al.  Clinically Available Optical Imaging Technologies in Endoscopic Lesion Detection: Current Status and Future Perspective , 2021, Journal of healthcare engineering.

[191]  Afaz Uddin Ahmed,et al.  Localization of Wireless Capsule Endoscope: A Systematic Review , 2017, IEEE Sensors Journal.

[192]  Guang-Zhong Yang,et al.  Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery , 2017, ArXiv.

[193]  Sebastian Bodenstedt,et al.  Dense GPU-enhanced surface reconstruction from stereo endoscopic images for intraoperative registration. , 2012, Medical physics.

[194]  David Barrett,et al.  Video-based computer navigation in knee arthroscopy for patient-specific ACL reconstruction , 2019, International Journal of Computer Assisted Radiology and Surgery.

[195]  Gauthier Lafruit,et al.  Cross-Based Local Stereo Matching Using Orthogonal Integral Images , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[196]  K. Mönkemüller,et al.  Present and future perspectives of virtual chromoendoscopy with i‐scan and optical enhancement technology , 2014, Digestive endoscopy : official journal of the Japan Gastroenterological Endoscopy Society.

[197]  Yunhui Liu,et al.  Vision-Based Calibration of Dual RCM-Based Robot Arms in Human-Robot Collaborative Minimally Invasive Surgery , 2018, IEEE Robotics and Automation Letters.

[198]  M. Iwamuro,et al.  Linked color imaging (LCI), a novel image-enhanced endoscopy technology, emphasizes the color of early gastric cancer , 2017, Endoscopy International Open.

[199]  Yan Ping Du,et al.  A Novel Endoscopic Navigation System: Simultaneous Endoscope and Radial Ultrasound Probe Tracking Without External Trackers , 2019, MICCAI.

[200]  Ignas Budvytis,et al.  Deep Multi-view Stereo for Dense 3D Reconstruction from Monocular Endoscopic Video , 2020, MICCAI.

[201]  W. Roberts,et al.  PD04-06 DEEP LEARNING COMPUTER VISION ALGORITHM FOR DETECTING KIDNEY STONE COMPOSITION: TOWARDS AN AUTOMATED FUTURE , 2019, Journal of Urology.

[202]  Ian D. Reid,et al.  Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[203]  Andreas Mandelis,et al.  Review of the state of the art in cardiovascular endoscopy imaging of atherosclerosis using photoacoustic techniques with pulsed and continuous-wave optical excitations , 2019, Journal of biomedical optics.

[204]  Faisal Mahmood,et al.  Deep learning and conditional random fields‐based depth estimation and topographical reconstruction from conventional endoscopy , 2017, Medical Image Anal..

[205]  Yasin Almalioglu,et al.  Unsupervised Odometry and Depth Learning for Endoscopic Capsule Robots , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[206]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.