Content-Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy

Fluorescence microscopy is a key driver of discoveries in the life-sciences, with observable phenomena being limited by the optics of the microscope, the chemistry of the fluorophores, and the maximum photon exposure tolerated by the sample. These limits necessitate trade-offs between imaging speed, spatial resolution, light exposure, and imaging depth. In this work we show how image restoration based on deep learning extends the range of biological phenomena observable by microscopy. On seven concrete examples we demonstrate how microscopy images can be restored even if 60-fold fewer photons are used during acquisition, how near isotropic resolution can be achieved with up to 10-fold under-sampling along the axial direction, and how tubular and granular structures smaller than the diffraction limit can be resolved at 20-times higher frame-rates compared to state-of-the-art methods. All developed image restoration methods are freely available as open source software in Python, Fiji, and Knime.

[1]  William H. Richardson,et al.  Bayesian-Based Iterative Method of Image Restoration , 1972 .

[2]  L. Lucy.  An iterative technique for the rectification of observed distributions , 1974 .

[3]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[4]  M. Gustafsson.  Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy , 2000, Journal of microscopy.

[5]  F. Del Bene,et al.  Optical Sectioning Deep Inside Live Embryos by Selective Plane Illumination Microscopy , 2004, Science.

[6]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[7]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[8]  J. Lippincott-Schwartz,et al.  Imaging Intracellular Fluorescent Proteins at Nanometer Resolution , 2006, Science.

[9]  Michael J Rust,et al.  Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM) , 2006, Nature Methods.

[10]  James B. Pawley,et al.  Fundamental Limits in Confocal Microscopy , 2006 .

[11]  Huanhuan Chen,et al.  Trade-Off Between Diversity and Accuracy in Ensemble Generation , 2006, Multi-Objective Machine Learning.

[12]  Susan J. Brown,et al.  The red flour beetle, Tribolium castaneum (Coleoptera): a model for studies of development and pest biology. , 2009, Cold Spring Harbor protocols.

[13]  M. Gustafsson,et al.  Subdiffraction resolution in continuous samples , 2009 .

[14]  Kostadin Dabov,et al.  BM3D Image Denoising with Shape-Adaptive Principal Component Analysis , 2009 .

[15]  H. Flyvbjerg,et al.  Optimized localization-analysis for single-molecule tracking and super-resolution microscopy , 2010, Nature Methods.

[16]  Frank Jülicher,et al.  Cell Flow Reorients the Axis of Planar Polarity in the Wing Epithelium of Drosophila , 2010, Cell.

[17]  M. Davidson,et al.  Rapid three-dimensional isotropic imaging of living cells using Bessel beam plane illumination , 2011, Nature Methods.

[18]  Ullrich Köthe,et al.  Ilastik: Interactive learning and segmentation toolkit , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[19]  Philipp J. Keller,et al.  Quantitative high-speed imaging of entire developing embryos with simultaneous multiview light-sheet microscopy , 2012, Nature Methods.

[20]  Shimon Weiss,et al.  Superresolution optical fluctuation imaging (SOFI). , 2012, Advances in experimental medicine and biology.

[21]  Eran Zahavy,et al.  Nano-Biotechnology for Biomedical and Diagnostic Research , 2012, Advances in Experimental Medicine and Biology.

[22]  Michael Pollard,et al.  High-resolution restoration of 3D structures from widefield images with extreme low signal-to-noise-ratio , 2013, Proceedings of the National Academy of Sciences.

[23]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[24]  Stephan Preibisch,et al.  Efficient Bayesian-based multiview deconvolution , 2013, Nature Methods.

[25]  Wesley R. Legant,et al.  Lattice light-sheet microscopy: Imaging molecules to embryos at high spatiotemporal resolution , 2014, Science.

[26]  Philipp J. Keller,et al.  Whole-animal functional and developmental imaging with isotropic spatial resolution , 2015, Nature Methods.

[27]  Kaoru Sugimura,et al.  Unified quantitative characterization of epithelial tissue development , 2015, eLife.

[28]  Corinna Blasse,et al.  Interplay of cell dynamics and epithelial tension during morphogenesis of the Drosophila pupal wing , 2015, eLife.

[29]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[30]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[31]  Y. Kalaidzidis,et al.  A versatile pipeline for the multi-scale digital reconstruction and quantitative analysis of 3D tissue architecture , 2015, eLife.

[32]  Yu-Bin Yang,et al.  Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections , 2016, NIPS.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Yu-Bin Yang,et al.  Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections , 2016, ArXiv.

[35]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[36]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[37]  Maurício Rocha-Martins,et al.  Independent modes of ganglion cell translocation ensure correct lamination of the zebrafish retina , 2016, bioRxiv.

[38]  Wolfgang Hübner,et al.  Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ , 2016, Nature Communications.

[39]  Eugene W. Myers,et al.  Adaptive light-sheet microscopy for long-term, high-resolution imaging in living organisms , 2016, Nature Biotechnology.

[40]  Ricardo Henriques,et al.  Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations , 2016, Nature Communications.

[41]  Radek Macháň,et al.  Multiple signal classification algorithm for super-resolution fluorescence microscopy , 2016, Nature Communications.

[42]  Frank Jülicher,et al.  TissueMiner: A multiscale analysis toolkit to quantify how cellular processes create tissue dynamics , 2016, eLife.

[43]  Lassi Paavolainen,et al.  Data-analysis strategies for image-based cell profiling , 2017, Nature Methods.

[44]  Yibo Zhang,et al.  Deep Learning Microscopy , 2017, ArXiv.

[45]  C. Norden,et al.  Phototoxicity in live fluorescence microscopy, and how to avoid it , 2017, BioEssays : news and reviews in molecular, cellular and developmental biology.

[46]  Nathalie Harder,et al.  An Objective Comparison of Cell Tracking Algorithms , 2017, Nature Methods.

[47]  Stephan Saalfeld,et al.  Deep Learning for Isotropic Super-Resolution from Non-isotropic 3D Electron Microscopy , 2017, MICCAI.

[48]  Fred A Hamprecht,et al.  Multicut brings automated neurite segmentation closer to human performance , 2017, Nature Methods.

[49]  Asm Shihavuddin,et al.  Smooth 2D manifold extraction from 3D image stack , 2017, Nature Communications.

[50]  Eugene W. Myers,et al.  PreMosa: extracting 2D surfaces from 3D microscopy mosaics , 2017, Bioinform..

[51]  Eugene W. Myers,et al.  Isotropic reconstruction of 3D fluorescence microscopy images using convolutional neural networks , 2017, MICCAI.

[52]  Dasaradhi Palakodeti,et al.  Hierarchies in light sensing and dynamic interactions between ocular and extraocular sensory networks in a flatworm , 2017, Science Advances.

[53]  Pavel Tomancak,et al.  Assessing phototoxicity in live fluorescence imaging , 2017, Nature Methods.

[54]  Tomer Michaeli,et al.  Deep-STORM: super-resolution single-molecule microscopy by deep learning , 2018, 1801.09631.

[55]  Christophe Zimmer,et al.  Deep learning massively accelerates super-resolution localization microscopy , 2018, Nature Biotechnology.

[56]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[57]  Mary M. Maleckar,et al.  Label-free prediction of three-dimensional fluorescence images from transmitted light microscopy , 2018 .