Automated Detection of Corneal Edema With Deep Learning-Assisted Second Harmonic Generation Microscopy

Second Harmonic Generation Microscopy (SHG) is widely acknowledged as a valuable non-linear optical imaging tool, its contrast mechanism providing the premises to non-invasively identify, characterize, and monitor changes in the collagen architecture of tissues. However, the interpretation of SHG data can pose difficulties even for experts histopathologists, which represents a bottleneck for the translation of SHG-based diagnostic frameworks to clinical settings. The use of artificial intelligence methods for automated SHG analysis is still in an early stage, with only few studies having been reported to date, none addressing ocular tissues yet. In this work we explore the use of three Deep Learning models, the highly popular InceptionV3 and ResNet50, alongside FLIMBA, a custom developed architecture, requiring no pre-training, to automatically detect corneal edema in SHG images of porcine cornea. We observe that Deep Learning models building on different architectures provide complementary results for the classification of cornea SHG images and demonstrate an AU-ROC = 0.98 for their joint use. These results have potential to be extrapolated to other diagnostics scenarios, such as automated extraction of hydration level of cornea, or identification of corneal edema causes, and thus pave the way for novel methods for precision diagnostics of the cornea with Deep-Learning assisted SHG imaging.

[1]  Rosa M. Martínez-Ojeda,et al.  Collagen Organization, Polarization Sensitivity and Image Quality in Human Corneas using Second Harmonic Generation Microscopy , 2022, Photonics.

[2]  S. Stanciu,et al.  PSHG-TISS: A collection of polarization-resolved second harmonic generation microscopy images of fixed tissues , 2022, Scientific data.

[3]  Daniel B. Zander,et al.  Corneal Oedema: Aetiology, Diagnostic Testing, and Treatment. , 2022, Klinische Monatsblatter fur Augenheilkunde.

[4]  A. Fletcher,et al.  Bespoke data augmentation and network construction enable developmental morphological classification on limited microscopy datasets , 2022, bioRxiv.

[5]  S. Zhuo,et al.  Machine learning-based rapid diagnosis of human borderline ovarian cancer on second-harmonic generation images. , 2021, Biomedical optics express.

[6]  A. Picón,et al.  Analysis on the Characterization of Multiphoton Microscopy Images for Malignant Neoplastic Colon Lesion Detection under Deep Learning Methods , 2021, Journal of pathology informatics.

[7]  B. Wilson,et al.  Machine learning-enabled cancer diagnostics with widefield polarimetric second-harmonic generation microscopy , 2021, Scientific Reports.

[8]  David W. Nauen,et al.  Label-free imaging of human brain tissue at subcellular resolution for potential rapid intra-operative assessment of glioma surgery , 2021, Theranostics.

[9]  Francisco J. Ávila,et al.  Assessment of the corneal collagen organization after chemical burn using second harmonic generation microscopy. , 2021, Biomedical optics express.

[10]  Raman Arora,et al.  On Convergence and Generalization of Dropout Training , 2020, NeurIPS.

[11]  Na Dong,et al.  Inception v3 based cervical cell classification combined with artificially extracted features , 2020, Appl. Soft Comput..

[12]  Emily G Pendleton,et al.  Second harmonic generation characterization of collagen in whole bone. , 2020, Biomedical optics express.

[13]  Zhenlin Zhan,et al.  Rapid identification of human ovarian cancer in second harmonic generation images using radiomics feature analyses and tree‐based pipeline optimization tool , 2020, Journal of biophotonics.

[14]  R. R. Sedamkar,et al.  Detecting Affect States Using VGG16, ResNet50 and SE-ResNet50 Networks , 2020, SN Computer Science.

[15]  Radu Hristu,et al.  Multiphoton microscopy of the dermoepidermal junction and automated identification of dysplastic tissues with deep learning. , 2019, Biomedical optics express.

[16]  Michael Schmitt,et al.  Automatic label‐free detection of breast cancer using nonlinear multimodal imaging and the convolutional neural network ResNet50 , 2019 .

[17]  Zheng Huang,et al.  Epithelium segmentation and automated Gleason grading of prostate cancer via deep learning in label‐free multiphoton microscopic images , 2019, Journal of biophotonics.

[18]  Yu Zeng,et al.  Pulmonary Image Classification Based on Inception-v3 Transfer Learning Model , 2019, IEEE Access.

[19]  Kevin McGuinness,et al.  Deep Learning for Computer Vision IMVIP 2019 : Irish Machine Vision and ImageProcessing 2019 Comparing Data Augmentation Strategies for Deep Image Classification , 2019 .

[20]  Zhiwen Yu,et al.  A survey on ensemble learning , 2019, Frontiers of Computer Science.

[21]  Francisco J. Ávila,et al.  In vivo two-photon microscopy of the human eye , 2019, Scientific Reports.

[22]  Sung-Jan Lin,et al.  Intravital multiphoton microscopic imaging platform for ocular surface imaging , 2019, Experimental eye research.

[23]  Pablo Artal,et al.  Quantitative Discrimination of Healthy and Diseased Corneas With Second Harmonic Generation Microscopy , 2019, Translational vision science & technology.

[24]  Ming Ni,et al.  Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning , 2019, Journal of biophotonics.

[25]  Michael Schmidt,et al.  Label‐Free Multiphoton Endomicroscopy for Minimally Invasive In Vivo Imaging , 2019, Advanced science.

[26]  Nick White,et al.  Quantification of collagen fiber structure using second harmonic generation imaging and two‐dimensional discrete Fourier transform analysis: Application to the human optic nerve head , 2019, Journal of biophotonics.

[27]  D. A. Vrazhnov,et al.  Analysis of Collagen Spatial Structure Using Multiphoton Microscopy and Machine Learning Methods , 2019, Biochemistry (Moscow).

[28]  D. Harris,et al.  Kinetics of corneal leukocytes by intravital multiphoton microscopy , 2018, FASEB journal : official publication of the Federation of American Societies for Experimental Biology.

[29]  Lior Rokach,et al.  Ensemble learning: A survey , 2018, WIREs Data Mining Knowl. Discov..

[30]  Robert W Boyd,et al.  Automated classification of multiphoton microscopy images of ovarian tissue using deep learning , 2018, Journal of biomedical optics.

[31]  Hossein Baharvand,et al.  Automatic white blood cell classification using pre-trained deep learning models: ResNet and Inception , 2018, International Conference on Machine Vision.

[32]  K. König State-of-the-art clinical multimodal multi photon / CARS / FLIM tomography of human skin (Conference Presentation) , 2018 .

[33]  Takeshi Imamura,et al.  Quantitative Morphometry for Osteochondral Tissues Using Second Harmonic Generation Microscopy and Image Texture Information , 2018, Scientific Reports.

[34]  Juan M. Bueno,et al.  Quantifying external and internal collagen organization from Stokes-vector-based second harmonic generation imaging polarimetry , 2017 .

[35]  Xiaoling Xia,et al.  Inception-v3 for flower classification , 2017, 2017 2nd International Conference on Image, Vision and Computing (ICIVC).

[36]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Karsten König,et al.  First multiphoton tomography of brain in man , 2016, SPIE BiOS.

[38]  Pablo Artal,et al.  Performance evaluation of a sensorless adaptive optics multiphoton microscope , 2016, Journal of microscopy.

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Cameron P. Brown,et al.  Analysis of forward and backward Second Harmonic Generation images to probe the nanoscale structure of collagen within bone and cartilage , 2015, Journal of biophotonics.

[41]  R. Chuck,et al.  Second Harmonic Generation Imaging Analysis of Collagen Arrangement in Human Cornea. , 2015, Investigative ophthalmology & visual science.

[42]  P. Artal,et al.  Second-harmonic generation microscopy of photocurable polymer intrastromal implants in ex-vivo corneas. , 2015, Biomedical optics express.

[43]  J. Jester,et al.  From nano to macro: studying the hierarchical structure of the corneal extracellular matrix. , 2015, Experimental eye research.

[44]  Charless C. Fowlkes,et al.  Do We Need More Training Data? , 2015, International Journal of Computer Vision.

[45]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Vikas Singh,et al.  Texture analysis applied to second harmonic generation image data for ovarian cancer classification , 2014, Journal of biomedical optics.

[47]  Wenhua Liu,et al.  Quantitative evaluation of skeletal muscle defects in second harmonic generation images , 2013, Journal of biomedical optics.

[48]  Roberto Pini,et al.  Thermal transitions of fibrillar collagen unveiled by second-harmonic generation microscopy of corneal stroma. , 2012, Biophysical journal.

[49]  Chen-Yuan Dong,et al.  Fast Fourier transform-based analysis of second-harmonic generation image in keratoconic cornea. , 2012, Investigative ophthalmology & visual science.

[50]  Raghu Ambekar,et al.  Quantitative second-harmonic generation microscopy for imaging porcine cortical bone: comparison to SEM and its potential to investigate age-related changes. , 2012, Bone.

[51]  Saeed Akhtar,et al.  Structure of corneal layers, collagen fibrils, and proteoglycans of tree shrew cornea , 2011, Molecular vision.

[52]  Chen-Yuan Dong,et al.  Second harmonic generation microscopy: principles and applications to disease diagnosis , 2011 .

[53]  N. Honarpisheh,et al.  Amantadine-associated corneal edema. , 2010, Parkinsonism & related disorders.

[54]  Shuangmu Zhuo,et al.  Quantitatively linking collagen alteration and epithelial tumor progression by second harmonic generation microscopy , 2010 .

[55]  C. Dong,et al.  Structural characterization of edematous corneas by forward and backward second harmonic generation imaging. , 2009, Biophysical journal.

[56]  Naoyuki Yamada,et al.  Detection of subepithelial fibrosis associated with corneal stromal edema by second harmonic generation imaging microscopy. , 2009, Investigative ophthalmology & visual science.

[57]  Hanry Yu,et al.  Fibro-C-Index: comprehensive, morphology-based quantification of liver fibrosis using second harmonic generation and two-photon microscopy. , 2009, Journal of biomedical optics.

[58]  Gerardo Hermosillo,et al.  Supervised learning from multiple experts: whom to trust when everyone lies a bit , 2009, ICML '09.

[59]  Watt W Webb,et al.  Optical visualization of Alzheimer's pathology via multiphoton-excited intrinsic fluorescence and second harmonic generation. , 2009, Optics express.

[60]  C. Dong,et al.  Second Harmonic Generation Microscopy Characterization of Corneal Edema , 2009 .

[61]  A. Ertan,et al.  Keratoconus Clinical Findings According to Different Age and Gender Groups , 2008, Cornea.

[62]  W. Wee,et al.  Corneal Endothelial Dysfunction Associated With Amantadine Toxicity , 2008, Cornea.

[63]  William A Mohler,et al.  Measurement of muscle disease by quantitative second-harmonic generation imaging. , 2008, Journal of biomedical optics.

[64]  Karsten König,et al.  Clinical multiphoton tomography , 2008, Journal of biophotonics.

[65]  Chen-Yuan Dong,et al.  Multiphoton fluorescence and second harmonic generation microscopy for imaging infectious keratitis. , 2007, Journal of biomedical optics.

[66]  Chen-Yuan Dong,et al.  Multiphoton fluorescence and second harmonic generation imaging of the structural alterations in keratoconus ex vivo. , 2006, Investigative ophthalmology & visual science.

[67]  M. C. Acosta,et al.  Influence of age, gender and iris color on mechanical and chemical sensitivity of the cornea and conjunctiva. , 2006, Experimental eye research.

[68]  B. Cho,et al.  Differences in corneal thickness and corneal endothelium related to duration in Diabetes , 2006, Eye.

[69]  William A Mohler,et al.  Characterization of the myosin-based source for second-harmonic generation from muscle sarcomeres. , 2006, Biophysical journal.

[70]  Leonardo Sacconi,et al.  Optical recording of fast neuronal membrane potential transients in acute mammalian brain slices by second-harmonic generation microscopy. , 2005, Journal of neurophysiology.

[71]  Chen-Yuan Dong,et al.  Characterizing the thermally induced structural changes to intact porcine eye, part 1: second harmonic generation imaging of cornea stroma. , 2005, Journal of biomedical optics.

[72]  S. Ricard-Blum,et al.  The collagen superfamily: from the extracellular matrix to the cell membrane. , 2005, Pathologie-biologie.

[73]  Josef Bille,et al.  Second harmonic generation imaging of collagen fibrils in cornea and sclera. , 2005, Optics express.

[74]  W. Webb,et al.  Nonlinear magic: multiphoton microscopy in the biosciences , 2003, Nature Biotechnology.

[75]  Watt W. Webb,et al.  Uniform polarity microtubule assemblies imaged in native brain tissue by second-harmonic generation microscopy , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[76]  Brian Seed,et al.  Dynamic imaging of collagen and its modulation in tumors in vivo using second-harmonic generation , 2003, Nature Medicine.

[77]  C. Kielty,et al.  The Collagen Family: Structure, Assembly, and Organization in the Extracellular Matrix , 2003 .

[78]  M. Dana,et al.  Corneal edema after cataract surgery: Incidence and etiology , 2002, Seminars in ophthalmology.

[79]  G. M. P. VAN KEMPEN,et al.  A quantitative comparison of image restoration methods for confocal microscopy , 1997 .

[80]  Max A. Viergever,et al.  Scale and the differential structure of images , 1992, Image Vis. Comput..

[81]  D A Newsome,et al.  Human corneal stroma contains three distinct collagens. , 1982, Investigative ophthalmology & visual science.

[82]  J. Levenson Corneal edema: cause and treatment. , 1975, Survey of ophthalmology.

[83]  D. Maurice The structure and transparency of the cornea , 1957, The Journal of physiology.

[84]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[85]  Abdul Kader Sagar,et al.  Second-harmonic generation imaging of cancer. , 2014, Methods in cell biology.

[86]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[87]  A. Worster,et al.  Understanding receiver operating characteristic (ROC) curves. , 2006, CJEM.

[88]  A. Robert,et al.  Corneal collagens. , 2001, Pathologie-biologie.

[89]  J R Beck,et al.  The use of relative operating characteristic (ROC) curves in test performance evaluation. , 1986, Archives of pathology & laboratory medicine.