Ultra-fast fit-free analysis of complex fluorescence lifetime imaging via deep learning

Fluorescence lifetime imaging (FLI) provides unique quantitative information in biomedical and molecular biology studies, but relies on complex data fitting techniques to derive the quantities of interest. Herein, we propose a novel fit-free approach in FLI image formation that is based on Deep Learning (DL) to quantify complex fluorescence decays simultaneously over a whole image and at ultra-fast speeds. Our deep neural network (DNN), named FLI-Net, is designed and model-based trained to provide all lifetime-based parameters that are typically employed in the field. We demonstrate the accuracy and generalizability of FLI-Net by performing quantitative microscopic and preclinical experimental lifetime-based studies across the visible and NIR spectra, as well as across the two main data acquisition technologies. Our results demonstrate that FLI-Net is well suited to quantify complex fluorescence lifetimes, accurately, in real time in cells and intact animals without any parameter settings. Hence, it paves the way to reproducible and quantitative lifetime studies at unprecedented speeds, for improved dissemination and impact of FLI in many important biomedical applications, especially in clinical settings.

[1]  Thomas Nowotny,et al.  Artificial neural network approaches for fluorescence lifetime imaging techniques. , 2016, Optics letters.

[2]  Raluca Niesner,et al.  Noniterative biexponential fluorescence lifetime imaging in the investigation of cellular metabolism by means of NAD(P)H autofluorescence. , 2004, Chemphyschem : a European journal of chemical physics and physical chemistry.

[3]  Yu Zhang,et al.  Very deep convolutional networks for end-to-end speech recognition , 2016, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[4]  R. Day,et al.  Investigating protein-protein interactions in living cells using fluorescence lifetime imaging microscopy , 2011, Nature Protocols.

[5]  W. Becker Advanced Time-Correlated Single Photon Counting Applications , 2015 .

[6]  Scott C Davis,et al.  Pre-clinical whole-body fluorescence imaging: Review of instruments, methods and applications. , 2010, Journal of photochemistry and photobiology. B, Biology.

[7]  E. Gratton,et al.  The phasor approach to fluorescence lifetime imaging analysis. , 2008, Biophysical journal.

[8]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[9]  Nicholas S. Peters,et al.  Characterization of NAD(P)H and FAD autofluorescence signatures in a Langendorff isolated-perfused rat heart model , 2018, Biomedical optics express.

[10]  X. Intes,et al.  Quantitative Imaging of Receptor-Ligand Engagement in Intact Live Animals , 2017, bioRxiv.

[11]  Loic A. Royer,et al.  Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy , 2018, bioRxiv.

[12]  Brian W. Pogue Optics in the Molecular Imaging Race , 2015 .

[13]  Farzad Fereidouni,et al.  A modified phasor approach for analyzing time‐gated fluorescence lifetime images , 2011, Journal of microscopy.

[14]  Xavier Intes,et al.  FLIM-FRET for Cancer Applications. , 2015, Current molecular imaging.

[15]  Xavier Intes,et al.  Assessment of Gate Width Size on Lifetime-Based Förster Resonance Energy Transfer Parameter Estimation , 2015, 2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC).

[16]  Enrico Gratton,et al.  Fit-free analysis of fluorescence lifetime imaging data using the phasor approach , 2018, Nature Protocols.

[17]  Evgeny Putin,et al.  Adversarial Threshold Neural Computer for Molecular de Novo Design. , 2018, Molecular pharmaceutics.

[18]  Tomer Michaeli,et al.  Deep-STORM: super-resolution single-molecule microscopy by deep learning , 2018, 1801.09631.

[19]  Pingkun Yan,et al.  Deep compressive macroscopic fluorescence lifetime imaging , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[20]  Alex J Walsh,et al.  Temporal binning of time-correlated single photon counting data improves exponential decay fits and imaging speed. , 2016, Biomedical optics express.

[21]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[22]  Sean C. Warren,et al.  Screening for protein-protein interactions using Förster resonance energy transfer (FRET) and fluorescence lifetime imaging microscopy (FLIM) , 2016, Scientific Reports.

[23]  Baran D. Sumer,et al.  A Transistor-like pH Nanoprobe for Tumour Detection and Image-guided Surgery , 2016, Nature Biomedical Engineering.

[24]  Yury Budansky,et al.  Alzheimer mouse brain tissue measured by time resolved fluorescence spectroscopy using single‐ and multi‐photon excitation of label free native molecules , 2018, Journal of biophotonics.

[25]  Bo Du,et al.  Deeply-supervised CNN for prostate segmentation , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[26]  N. Ramanujam,et al.  In vivo multiphoton microscopy of NADH and FAD redox states, fluorescence lifetimes, and cellular morphology in precancerous epithelia , 2007, Proceedings of the National Academy of Sciences.

[27]  Alix Le Marois,et al.  Fluorescence lifetime imaging (Flim): Basic concepts and recent applications , 2015 .

[28]  Jiajun Zhang,et al.  Deep Neural Networks in Machine Translation: An Overview , 2015, IEEE Intelligent Systems.

[29]  Laura Marcu,et al.  Percutaneous fiber-optic sensor for chronic glucose monitoring in vivo. , 2008, Biosensors & bioelectronics.

[30]  Petr Herman,et al.  Fluorescence lifetime‐resolved pH imaging of living cells , 2003, Cytometry. Part A : the journal of the International Society for Analytical Cytology.

[31]  Xavier Intes,et al.  Comparison of illumination geometry for lifetime‐based measurements in whole‐body preclinical imaging , 2018, Journal of biophotonics.

[32]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[33]  Xavier Intes,et al.  Development of an optical imaging platform for functional imaging of small animals using wide-field excitation , 2010, Biomedical optics express.

[34]  X. Intes,et al.  In vitro and in vivo phasor analysis of stoichiometry and pharmacokinetics using near-infrared dyes , 2018, bioRxiv.

[35]  Martin Hammer,et al.  Review of clinical approaches in fluorescence lifetime imaging ophthalmoscopy , 2018, Journal of biomedical optics.

[36]  W. Becker Fluorescence lifetime imaging – techniques and applications , 2012, Journal of microscopy.

[37]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[38]  Seok Hyun Yun,et al.  Light in diagnosis, therapy and surgery , 2016, Nature Biomedical Engineering.

[39]  Christophe Zimmer,et al.  Deep learning massively accelerates super-resolution localization microscopy , 2018, Nature Biotechnology.

[40]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Chandra Shekhar,et al.  On simplified application of multidimensional Savitzky-Golay filters and differentiators , 2016 .

[42]  Xavier Intes,et al.  Non-Invasive In Vivo Imaging of Near Infrared-labeled Transferrin in Breast Cancer Cells and Tumors Using Fluorescence Lifetime FRET , 2013, PloS one.

[43]  Xavier Intes,et al.  Reduced temporal sampling effect on accuracy of time-domain fluorescence lifetime Förster resonance energy transfer. , 2014, Journal of biomedical optics.

[44]  Jiong Ma,et al.  Rapid diagnosis and intraoperative margin assessment of human lung cancer with fluorescence lifetime imaging microscopy , 2017, BBA clinical.

[45]  Andrew H. Beck,et al.  Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer , 2017, JAMA.

[46]  Mary M. Maleckar,et al.  Label-free prediction of three-dimensional fluorescence images from transmitted light microscopy , 2018 .

[47]  Bastian Leibe,et al.  Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Asima Pradhan,et al.  Spatio‐temporal map for early cancer detection: Proof of concept , 2018, Journal of biophotonics.

[49]  Neil A. Thacker,et al.  The Bhattacharyya metric as an absolute similarity measure for frequency coded data , 1998, Kybernetika.

[50]  Yibo Zhang,et al.  Deep Learning Microscopy , 2017, ArXiv.

[51]  Enrico Gratton,et al.  Fluorescence lifetime imaging of endogenous biomarker of oxidative stress , 2015, Scientific Reports.

[52]  Xavier Intes,et al.  Fluorescence lifetime FRET imaging of receptor-ligand complexes in tumor cells in vitro and in vivo , 2017, BiOS.

[53]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[54]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[55]  Stephen T. C. Wong,et al.  Combining deep learning and coherent anti-Stokes Raman scattering imaging for automated differential diagnosis of lung cancer , 2017, Journal of biomedical optics.

[56]  Alex J Walsh,et al.  Optical metabolic imaging identifies glycolytic levels, subtypes, and early-treatment response in breast cancer. , 2013, Cancer research.

[57]  X. Intes,et al.  In vitro and in vivo phasor analysis of stoichiometry and pharmacokinetics using short‐lifetime near‐infrared dyes and time‐gated imaging , 2018, Journal of biophotonics.