Applications, Promises, and Pitfalls of Deep Learning for Fluorescence Image Reconstruction

Deep Learning is a recent and important addition to the computational toolbox available for image reconstruction in fluorescence microscopy. We review state-of-the-art applications such as image restoration, super-resolution, and light-field imaging, and discuss how the latest Deep Learning research can be applied to other image reconstruction tasks such as structured illumination, spectral deconvolution, and sample stabilisation. Despite its successes, Deep Learning also poses significant challenges, has often misunderstood capabilities, and overlooked limits. We will address key questions, such as: What are the challenges in obtaining training data? Can we discover structures not present in the training data? And, what is the danger of inferring unsubstantiated image details?

[1]  A. Cardona,et al.  Elastic volume reconstruction from series of ultra-thin microscopy sections , 2012, Nature Methods.

[2]  Aggelos K. Katsaggelos,et al.  Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods , 2018, IEEE Signal Processing Magazine.

[3]  E. Boyden,et al.  Simultaneous whole-animal 3D-imaging of neuronal activity using light-field microscopy , 2014, Nature Methods.

[4]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[5]  Thierry Blu,et al.  Image Denoising in Mixed Poisson–Gaussian Noise , 2011, IEEE Transactions on Image Processing.

[6]  Quanshi Zhang,et al.  Interpreting CNN knowledge via an Explanatory Graph , 2017, AAAI.

[7]  Yi Li,et al.  Network-based instantaneous recording and video-rate reconstruction of 4D biological dynamics , 2018, bioRxiv.

[8]  Elliot M. Meyerowitz,et al.  Observing the cell in its native state: Imaging subcellular dynamics in multicellular organisms , 2018, Science.

[9]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[10]  Jürgen Schmidhuber,et al.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction , 2011, ICANN.

[11]  M. Davidson,et al.  Extended-resolution structured illumination imaging of endocytic and cytoskeletal dynamics , 2015, Science.

[12]  Aydogan Ozcan,et al.  Deep learning achieves super-resolution in fluorescence microscopy , 2018, bioRxiv.

[13]  Charles Kervrann,et al.  Fast live simultaneous multiwavelength four-dimensional optical microscopy , 2010, Proceedings of the National Academy of Sciences.

[14]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Huafeng Liu,et al.  Untwisting the Caenorhabditis elegans embryo , 2015, eLife.

[16]  David J. Fleet,et al.  Adversarial Manipulation of Deep Representations , 2015, ICLR.

[17]  Stephan Preibisch,et al.  Efficient Bayesian-based multiview deconvolution , 2013, Nature Methods.

[18]  Vasilis Ntziachristos,et al.  In vivo imaging of Drosophila melanogaster pupae with mesoscopic fluorescence tomography , 2007, Nature Methods.

[19]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[20]  Eugene W. Myers,et al.  Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks , 2017, MICCAI.

[21]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[22]  Josiane Zerubia,et al.  Richardson–Lucy algorithm with total variation regularization for 3D confocal microscope deconvolution , 2006, Microscopy research and technique.

[23]  Elsa D. Angelini,et al.  A Compressed Sensing Approach for Biological Microscopy Image Denoising , 2009 .

[25]  Ricardo Henriques,et al.  Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations , 2016, Nature Communications.

[26]  Eugene W. Myers,et al.  PreMosa: extracting 2D surfaces from 3D microscopy mosaics , 2017, Bioinform..

[27]  Sarah Webb.  Deep learning for biology , 2018, Nature.

[28]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[29]  Stefan Roth,et al.  UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss , 2017, AAAI.

[30]  Tomer Michaeli,et al.  Multi-scale Weighted Nuclear Norm Image Restoration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Loic A. Royer,et al.  Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy , 2018, bioRxiv.

[32]  Casper F Winsnes,et al.  Deep learning is combined with massive-scale citizen science to improve large-scale image classification , 2018, Nature Biotechnology.

[33]  Yann LeCun,et al.  Computing the stereo matching cost with a convolutional neural network , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Fernando Amat,et al.  Efficient processing and analysis of large-scale light-sheet microscopy data , 2015, Nature Protocols.

[35]  Vijay Kumar,et al.  Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model , 2017, IEEE Robotics and Automation Letters.

[36]  R. Dobarzić,et al.  [Fluorescence microscopy]. , 1975, Plucne bolesti i tuberkuloza.

[37]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[38]  Franck Marchis,et al.  AIDA: Adaptive Image Deconvolution Algorithm , 2013 .

[39]  Florian Jug,et al.  Cryo-CARE: Content-Aware Image Restoration for Cryo-Transmission Electron Microscopy Data , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[40]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[41]  H. Sebastian Seung,et al.  Natural Image Denoising with Convolutional Networks , 2008, NIPS.

[42]  Bruce R. Rosen,et al.  Image reconstruction by domain-transform manifold learning , 2017, Nature.

[43]  Benjamin Recht,et al.  Do CIFAR-10 Classifiers Generalize to CIFAR-10? , 2018, ArXiv.

[44]  Jaakko Lehtinen,et al.  Noise2Noise: Learning Image Restoration without Clean Data , 2018, ICML.

[45]  J. Sharpe,et al.  Attenuation artifacts in light sheet fluorescence microscopy corrected by OPTiSPIM , 2018, Light: Science & Applications.

[46]  Loïc Royer,et al.  Noise2Self: Blind Denoising by Self-Supervision , 2019, ICML.

[47]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  M. Hutson.  Artificial intelligence faces reproducibility crisis. , 2018, Science.

[49]  Patrick Bouthemy,et al.  Patch-Based Nonlocal Functional for Denoising Fluorescence Microscopy Image Sequences , 2010, IEEE Transactions on Medical Imaging.

[50]  Laura Waller,et al.  DiffuserCam: Lensless Single-exposure 3D Imaging , 2017, ArXiv.

[51]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[52]  Aydogan Ozcan,et al.  Deep Learning Microscopy: Enhancing Resolution, Field-of-View and Depth-of-Field of Optical Microscopy Images Using Neural Networks , 2017, 2018 Conference on Lasers and Electro-Optics (CLEO).

[53]  Anne Sentenac,et al.  Structured illumination microscopy using unknown speckle patterns , 2012, Nature Photonics.

[54]  Michael J Rust,et al.  Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM) , 2006, Nature Methods.

[55]  Eugene W. Myers,et al.  Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms , 2016, Nature Biotechnology.

[56]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.

[57]  Philip Bachman,et al.  Deep Reinforcement Learning that Matters , 2017, AAAI.

[58]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[59]  Kazuhiro Terao,et al.  Machine learning at the energy and intensity frontiers of particle physics , 2018, Nature.

[60]  Yan Xu,et al.  Deep learning of feature representation with multiple instance learning for medical image analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[61]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[62]  Nalan Liv,et al.  Automated sub-5 nm image registration in integrated correlative fluorescence and electron microscopy using cathodoluminescence pointers , 2017, Scientific Reports.

[63]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  M. Gustafsson,et al.  Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination. , 2008, Biophysical journal.

[65]  Federico Vaggi,et al.  GANs for Biological Image Synthesis , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[66]  Chin-Hui Lee,et al.  Convolutional-Recurrent Neural Networks for Speech Enhancement , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[67]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[68]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[69]  Dacheng Tao,et al.  DehazeNet: An End-to-End System for Single Image Haze Removal , 2016, IEEE Transactions on Image Processing.

[70]  Lei Zhang,et al.  Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising , 2016, IEEE Transactions on Image Processing.

[71]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[72]  Benjamin Recht,et al.  DeepLoco: Fast 3D Localization Microscopy Using Neural Networks , 2018, bioRxiv.

[73]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  R. Heintzmann,et al.  Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIM algorithm. , 2013, Optics letters.

[75]  Jaakko Lehtinen,et al.  Self-Supervised Deep Image Denoising , 2019, ArXiv.

[76]  David Zhang,et al.  A Trilateral Weighted Sparse Coding Scheme for Real-World Image Denoising , 2018, ECCV.

[77]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[78]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[79]  Quanshi Zhang,et al.  Interpretable Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[80]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[81]  Mary M. Maleckar,et al.  Label-free prediction of three-dimensional fluorescence images from transmitted light microscopy , 2018 .

[82]  Benjamin Schmid,et al.  Hyperspectral light sheet microscopy , 2015, Nature Communications.

[83]  Karen O. Egiazarian,et al.  Image denoising with block-matching and 3D filtering , 2006, Electronic Imaging.

[84]  David Kipping,et al.  A machine learns to predict the stability of circumbinary planets , 2018, 1801.03955.

[85]  C. Zimmer,et al.  QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ , 2010, Nature Methods.

[86]  Michael Liebling,et al.  Semi-Blind Spatially-Variant Deconvolution in Optical Microscopy with Local Point Spread Function Estimation by Use of Convolutional Neural Networks , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[87]  Tomer Michaeli,et al.  Deep-STORM: super-resolution single-molecule microscopy by deep learning , 2018, 1801.09631.

[88]  Christophe Zimmer,et al.  Deep learning massively accelerates super-resolution localization microscopy , 2018, Nature Biotechnology.

[89]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[90]  J. Lippincott-Schwartz,et al.  Imaging Intracellular Fluorescent Proteins at Nanometer Resolution , 2006, Science.

[91]  Francesco Cutrale,et al.  Hyperspectral phasor analysis enables multiplexed 5D in vivo imaging , 2017, Nature Methods.

[92]  Anne E Carpenter,et al.  Annotated high-throughput microscopy image sets for validation , 2012, Nature Methods.

[93]  Maxime Sermesant,et al.  SVF-Net: Learning Deformable Image Registration Using Shape Matching , 2017, MICCAI.

[94]  S. Grill,et al.  Active torque generation by the actomyosin cell cortex drives left–right symmetry breaking , 2014, eLife.

[95]  Sebastian Thrun,et al.  Dermatologist-level classification of skin cancer with deep neural networks , 2017, Nature.