Deep learning-enhanced light-field imaging with continuous validation

Light-field microscopy (LFM) has emerged as a powerful tool for fast volumetric image acquisition in biology, but its effective throughput and widespread use has been hampered by a computationally demanding and artefact-prone image reconstruction process. Here, we present a novel framework consisting of a hybrid light-field light-sheet microscope and deep learning-based volume reconstruction, where single light-sheet acquisitions continuously serve as training data and validation for the convolutional neural network reconstructing the LFM volume. Our network delivers high-quality reconstructions at video-rate throughput and we demonstrate the capabilities of our approach by imaging medaka heart dynamics and zebrafish neural activity.

[1]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[2]  F. Del Bene,et al.  Optical Sectioning Deep Inside Live Embryos by Selective Plane Illumination Microscopy , 2004, Science.

[3]  Aaron S. Andalman,et al.  Wave optics theory and 3-D deconvolution for the light field microscope. , 2013, Optics express.

[4]  Haoyu Li,et al.  Fourier light-field microscopy. , 2019, Optics express.

[5]  A. Vaziri,et al.  Video rate volumetric Ca2+ imaging across cortex using seeded iterative demixing (SID) microscopy , 2017, Nature Methods.

[6]  Hari Shroff,et al.  Faster fluorescence microscopy: advances in high speed biological imaging. , 2014, Current opinion in chemical biology.

[7]  Tobias Nöbauer,et al.  Brain-wide 3D light-field imaging of neuronal activity with speckle-enhanced resolution , 2018 .

[8]  Stephan Preibisch,et al.  Efficient Bayesian-based multiview deconvolution , 2013, Nature Methods.

[9]  Tobias Lasser,et al.  Artifact-free deconvolution in light field microscopy. , 2019, Optics express.

[10]  J. Wittbrodt,et al.  Medaka spalt acts as a target gene of hedgehog signaling. , 1997, Development.

[11]  Daniel E. S. Koo,et al.  High-contrast, synchronous volumetric imaging with selective volume illumination microscopy , 2020, Communications Biology.

[12]  Loic A. Royer,et al.  Applications, Promises, and Pitfalls of Deep Learning for Fluorescence Image Reconstruction , 2018 .

[13]  Qionghai Dai,et al.  DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy , 2019, Biophotonics Congress: Optics in the Life Sciences Congress 2019 (BODA,BRAIN,NTM,OMA,OMP).

[14]  Marc Levoy,et al.  Light field microscopy , 2006, ACM Trans. Graph..

[15]  Loic A. Royer,et al.  Content-aware image restoration: pushing the limits of fluorescence microscopy , 2018, Nature Methods.

[16]  Hirofumi Kobayashi,et al.  Image Deconvolution via Noise-Tolerant Self-Supervised Inversion , 2020, ArXiv.

[17]  E. Boyden,et al.  Simultaneous whole-animal 3D-imaging of neuronal activity using light-field microscopy , 2014, Nature Methods.

[18]  Oliver Skocek,et al.  Video rate volumetric Ca2+ imaging across cortical layers using Seeded Iterative Demixing (SID) microscopy , 2017, bioRxiv.

[19]  J. Wittbrodt,et al.  Transgenesis in fish: efficient selection of transgenic fish by co-injection with a fluorescent reporter construct , 2006, Nature Protocols.

[20]  Benjamin Schmid,et al.  Rapid 3D light-sheet microscopy with a tunable lens. , 2013, Optics express.

[21]  Florian Jug,et al.  Noise2Void - Learning Denoising From Single Noisy Images , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Michael Broxton,et al.  Fast near-whole–brain imaging in adult Drosophila during responses to stimuli and behavior , 2015, bioRxiv.

[23]  Stephanie C. Seeman,et al.  Bright and photostable chemigenetic indicators for extended in vivo voltage imaging , 2018, Science.

[24]  Huafeng Liu,et al.  Rapid image deconvolution and multiview fusion for optical microscopy , 2020, Nature Biotechnology.

[25]  Nils Wagner,et al.  Instantaneous isotropic volumetric imaging of fast biological processes , 2019, Nature Methods.

[26]  G Saavedra,et al.  FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples. , 2018, Biomedical optics express.

[27]  Paolo Favaro,et al.  Learning to Reconstruct Confocal Microscopy Stacks From Single Light Field Images , 2020, IEEE Transactions on Computational Imaging.

[28]  Yichen Ding,et al.  Network-based instantaneous recording and video-rate reconstruction of 4D biological dynamics , 2019, bioRxiv.

[29]  Aaron S. Andalman,et al.  Enhancing the performance of the light field microscope using wavefront coding. , 2014, Optics express.

[30]  Johannes D. Seelig,et al.  Video-rate volumetric functional imaging of the brain at synaptic resolution , 2016, Nature Neuroscience.

[31]  Bernhard Schölkopf,et al.  Learning to Deblur , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.