Automated video-mosaicking approach for confocal microscopic imaging in vivo: an approach to address challenges in imaging living tissue and extend field of view

We describe a computer vision-based mosaicking method for in vivo videos of reflectance confocal microscopy (RCM). RCM is a microscopic imaging technique, which enables the users to rapidly examine tissue in vivo. Providing resolution at cellular-level morphology, RCM imaging combined with mosaicking has shown to be highly sensitive and specific for non-invasively guiding skin cancer diagnosis. However, current RCM mosaicking techniques with existing microscopes have been limited to two-dimensional sequences of individual still images, acquired in a highly controlled manner, and along a specific predefined raster path, covering a limited area. The recent advent of smaller handheld microscopes is enabling acquisition of videos, acquired in a relatively uncontrolled manner and along an ad-hoc arbitrarily free-form, non-rastered path. Mosaicking of video-images (video-mosaicking) is necessary to display large areas of tissue. Our video-mosaicking methods addresses this need. The method can handle unique challenges encountered during video capture such as motion blur artifacts due to rapid motion of the microscope over the imaged area, warping in frames due to changes in contact angle and varying resolution with depth. We present test examples of video-mosaics of melanoma and non-melanoma skin cancers, to demonstrate potential clinical utility.

[1]  F Piccinini,et al.  Automated image mosaics by non‐automated light microscopes: the MicroMos software tool , 2013, Journal of microscopy.

[2]  M. Oliviero,et al.  Accuracy of in vivo confocal microscopy for diagnosis of basal cell carcinoma: a comparative study between handheld and wide‐probe confocal imaging , 2015, Journal of the European Academy of Dermatology and Venereology : JEADV.

[3]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[4]  Robert G. Moore,et al.  Initial clinical experience , 1997 .

[5]  R S Balaban,et al.  Rapid overlapping‐volume acquisition and reconstruction (ROVAR): automated 3D tiling for high‐resolution, large field‐of‐view optical microscopy , 2011, Journal of microscopy.

[6]  C. Longo,et al.  Integration of reflectance confocal microscopy in sequential dermoscopy follow‐up improves melanoma detection accuracy , 2015, The British journal of dermatology.

[7]  Catherine E Scarff,et al.  Reflectance confocal microscopy of cutaneous tumors: an atlas with clinical, dermoscopic and histological correlations , 2009 .

[8]  Nicholas Ayache,et al.  Robust mosaicing with correction of motion distortions and tissue deformations for in vivo fibered microscopy , 2006, Medical Image Anal..

[9]  Wei-Yen Hsu,et al.  Automatic seamless mosaicing of microscopic images: enhancing appearance with colour degradation compensation and wavelet‐based blending , 2008, Journal of microscopy.

[10]  G. Argenziano,et al.  Dermoscopic difficult lesions: an objective evaluation of reflectance confocal microscopy impact for accurate diagnosis , 2015, Journal of the European Academy of Dermatology and Venereology : JEADV.

[11]  Zachary M. Eastman,et al.  Confocal mosaicing microscopy in Mohs skin excisions: feasibility of rapid surgical pathology. , 2008, Journal of biomedical optics.

[12]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[13]  R. Hofmann-Wellenhof,et al.  Impact of in vivo reflectance confocal microscopy on the number needed to treat melanoma in doubtful lesions , 2014 .

[14]  智一 吉田,et al.  Efficient Graph-Based Image Segmentationを用いた圃場図自動作成手法の検討 , 2014 .

[15]  D. Kadouch,et al.  Treatment of Basal Cell Carcinoma Using a One-Stop-Shop With Reflectance Confocal Microscopy: Study Design and Protocol of a Randomized Controlled Multicenter Trial , 2015, JMIR research protocols.

[16]  D H Brooks,et al.  Video‐mosaicing of reflectance confocal images for examination of extended areas of skin in vivo , 2014, The British journal of dermatology.

[17]  James G. Fujimoto,et al.  Assessment of breast pathologies using nonlinear microscopy , 2014, Proceedings of the National Academy of Sciences.

[18]  A. Scope,et al.  Use of handheld reflectance confocal microscopy for in vivo diagnosis of solitary facial papules: a case series , 2014, Journal of the European Academy of Dermatology and Venereology : JEADV.

[19]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[20]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[21]  T Meyer,et al.  Seamless stitching of tile scan microscope images , 2015, Journal of microscopy.

[22]  Susan Mallett,et al.  Systematic review of diagnostic accuracy of reflectance confocal microscopy for melanoma diagnosis in patients with clinically equivocal skin lesions , 2013, Dermatology practical & conceptual.

[23]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[24]  L. Trümper From bench to bedside , 2005, Medizinische Klinik.

[25]  A. Marghoob,et al.  Reflectance confocal microscopy of skin in vivo: From bench to bedside , 2017, Lasers in surgery and medicine.

[26]  M E Martone,et al.  Automated microscopy system for mosaic acquisition and processing , 2006, Journal of microscopy.

[27]  Miguel Cordova,et al.  Intraoperative imaging during Mohs surgery with reflectance confocal microscopy: initial clinical experience , 2015, Journal of biomedical optics.

[28]  Shaoqun Zeng,et al.  Rapid imaging of large tissues using high-resolution stage-scanning microscopy. , 2015, Biomedical optics express.

[29]  J. Malvehy,et al.  In vivo reflectance confocal microscopy of equivocal melanocytic lesions detected by digital dermoscopy follow‐up , 2015, Journal of the European Academy of Dermatology and Venereology : JEADV.

[30]  Tomasz S. Tkaczyk,et al.  Real-time video mosaicing with a high-resolution microendoscope , 2012, Biomedical optics express.

[31]  Kristen C. Maitland,et al.  Imaging inflammation in mouse colon using a rapid stage-scanning confocal fluorescence microscope. , 2012, Journal of biomedical optics.

[32]  Y. Xiong,et al.  A meta‐analysis of reflectance confocal microscopy for the diagnosis of malignant skin tumours , 2016, Journal of the European Academy of Dermatology and Venereology : JEADV.

[33]  J. Fujimoto,et al.  Rapid imaging of surgical breast excisions using direct temporal sampling two photon fluorescent lifetime imaging. , 2015, Biomedical optics express.

[34]  C. Longo,et al.  Reflectance confocal microscopy as a second‐level examination in skin oncology improves diagnostic accuracy and saves unnecessary excisions: a longitudinal prospective study , 2014, The British journal of dermatology.

[35]  B. Preston,et al.  Case Series , 2010, Toxicologic pathology.

[36]  John Kenneth Salisbury,et al.  In Vivo Micro-Image Mosaicing , 2011, IEEE Transactions on Biomedical Engineering.

[37]  Liang Gao,et al.  Extend the field of view of selective plan illumination microscopy by tiling the excitation light sheet. , 2015, Optics express.

[38]  M. Rajadhyaksha,et al.  Correlation of Handheld Reflectance Confocal Microscopy With Radial Video Mosaicing for Margin Mapping of Lentigo Maligna and Lentigo Maligna Melanoma , 2017, JAMA dermatology.

[39]  VekslerOlga,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001 .

[40]  James W. Patterson,et al.  Skin Mapping With Punch Biopsies for Defining Margins in Melanoma: When You Don’t Know How Far to Go , 2008, Annals of Surgical Oncology.

[41]  Irfan A. Essa,et al.  Graphcut textures: image and video synthesis using graph cuts , 2003, ACM Trans. Graph..

[42]  C H A N G M I N G S U N,et al.  Mosaicing of microscope images with global geometric and radiometric corrections , 2005 .

[43]  Miguel Cordova,et al.  Handheld Reflectance Confocal Microscopy for the Detection of Recurrent Extramammary Paget Disease , 2017, JAMA dermatology.

[44]  David Salesin,et al.  Interactive digital photomontage , 2004, ACM Trans. Graph..

[45]  Milind Rajadhyaksha,et al.  Rapid confocal imaging of large areas of excised tissue with strip mosaicing. , 2011, Journal of biomedical optics.