Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks

Digital holography is one of the most widely used label-free microscopy techniques in biomedical imaging. Recovery of the missing phase information of a hologram is an important step in holographic image reconstruction. Here we demonstrate a convolutional recurrent neural network (RNN) based phase recovery approach that uses multiple holograms, captured at different sample-to-sensor distances to rapidly reconstruct the phase and amplitude information of a sample, while also performing autofocusing through the same network. We demonstrated the success of this deep learning-enabled holography method by imaging microscopic features of human tissue samples and Papanicolaou (Pap) smears. These results constitute the first demonstration of the use of recurrent neural networks for holographic imaging and phase recovery, and compared with existing methods, the presented approach improves the reconstructed image quality, while also increasing the depth-of-field and inference speed.

[1]  Wolfgang Osten,et al.  Resolution improvement in digital holography by angular and polarization multiplexing. , 2011, Applied optics.

[2]  S. Wilkins,et al.  Linear algorithms for phase retrieval in the Fresnel region , 2004 .

[3]  R Riesenberg,et al.  Quantitative phase and refractive index measurements with point-source digital in-line holographic microscopy. , 2012, Applied optics.

[4]  J. Samson,et al.  A Submersible Holographic Microscope for 4-D In-Situ Studies of Micro-Organisms in the Ocean with Intensity and Quantitative Phase Imaging , 2015 .

[5]  Sang Joon Lee,et al.  Deep learning-based digital in-line holographic microscopy for high resolution with extended field of view , 2019, Optics & Laser Technology.

[6]  Qian Kemao,et al.  Y-Net: a one-to-two deep learning framework for digital holographic reconstruction. , 2019, Optics letters.

[7]  Pasquale Memmolo,et al.  Tomographic flow cytometry by digital holography , 2016, Light: Science & Applications.

[8]  Yibo Zhang,et al.  Wide-field computational imaging of pathology slides using lens-free on-chip microscopy , 2014, Science Translational Medicine.

[9]  Kevin de Haan,et al.  Deep learning-based holographic polarization microscopy , 2020, ACS photonics.

[10]  G. Pedrini,et al.  Lensless phase microscopy using phase retrieval with multiple illumination wavelengths. , 2012, Applied optics.

[11]  Yibo Zhang,et al.  Deep learning-based super-resolution in coherent imaging systems , 2018, Scientific Reports.

[12]  Aydogan Ozcan,et al.  High-throughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories , 2012, Proceedings of the National Academy of Sciences.

[13]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[14]  A. Ozcan,et al.  Ultra wide-field lens-free monitoring of cells on-chip. , 2008, Lab on a chip.

[15]  A. Ozcan,et al.  Synthetic aperture-based on-chip microscopy , 2015, Light: Science & Applications.

[16]  P. Ferraro,et al.  Microscopy imaging and quantitative phase contrast mapping in turbid microfluidic channels by digital holography. , 2012, Lab on a chip.

[17]  J. van Leeuwen,et al.  Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.

[18]  Guohai Situ,et al.  eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction. , 2018, Optics express.

[19]  M. Teague.  Deterministic phase retrieval: a Green’s function solution , 1983 .

[20]  Aydogan Ozcan,et al.  Lensless digital holographic microscopy and its applications in biomedicine and environmental monitoring. , 2017, Methods.

[21]  Lutz Prechelt,et al.  Early Stopping-But When? , 1996, Neural Networks: Tricks of the Trade.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Aydogan Ozcan,et al.  Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram , 2018, Light: Science & Applications.

[24]  Bahram Javidi,et al.  Compact and field-portable 3D printed shearing digital holographic microscope for automated cell identification. , 2017, Applied optics.

[25]  Aydogan Ozcan,et al.  Lensless Imaging and Sensing. , 2016, Annual review of biomedical engineering.

[26]  L. Tian,et al.  Quantitative differential phase contrast imaging in an LED array microscope. , 2015, Optics express.

[27]  B Y Gu,et al.  Gerchberg-Saxton and Yang-Gu algorithms for phase retrieval in a nonunitary transform system: a comparison. , 1994, Applied optics.

[28]  A. Ozcan,et al.  Deep learning in holography and coherent imaging , 2019, Light: Science & Applications.

[29]  Bahram Javidi,et al.  Quantitative phase-contrast imaging with compact digital holographic microscope employing Lloyd's mirror. , 2012, Optics letters.

[30]  Aydogan Ozcan,et al.  Edge sparsity criterion for robust holographic autofocusing. , 2017, Optics letters.

[31]  Leslie J. Allen,et al.  Phase retrieval from series of images obtained by defocus variation , 2001 .

[32]  A. Ozcan,et al.  Maskless imaging of dense samples using pixel super-resolution based multi-height lensfree on-chip microscopy , 2012, Optics Express.

[33]  Yibo Zhang,et al.  Sparsity-based multi-height phase recovery in holographic microscopy , 2016, Scientific Reports.

[34]  George Barbastathis,et al.  Low Photon Count Phase Retrieval Using Deep Learning. , 2018, Physical review letters.

[35]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[36]  Vittorio Bianco,et al.  A deep learning-enabled portable imaging flow cytometer for cost-effective, high-throughput, and label-free analysis of natural water samples , 2018, Light: Science & Applications.

[37]  Natan T Shaked,et al.  Dual-interference-channel quantitative-phase microscopy of live cell dynamics. , 2009, Optics letters.

[38]  Aydogan Ozcan,et al.  Wide-field computational color imaging using pixel super-resolved on-chip microscopy. , 2013, Optics express.

[39]  S Marchesini,et al.  Invited article: a [corrected] unified evaluation of iterative projection algorithms for phase retrieval. , 2006, The Review of scientific instruments.

[40]  Derek Tseng,et al.  Compact, light-weight and cost-effective microscope based on lensless incoherent holography for telemedicine applications. , 2010, Lab on a chip.

[41]  A. Ozcan,et al.  On the use of deep learning for computational imaging , 2019, Optica.

[42]  Derek Tseng,et al.  Lensfree microscopy on a cellphone. , 2010, Lab on a chip.

[43]  Demetri Psaltis,et al.  Three-dimensional harmonic holographic microcopy using nanoparticles as probes for cell imaging. , 2009, Optics express.

[44]  Gabriel Popescu,et al.  Real Time Blood Testing Using Quantitative Phase Imaging , 2013, PloS one.

[45]  Demetri Psaltis,et al.  Coherent anti-Stokes Raman holography for chemically selective single-shot nonscanning 3D imaging. , 2010, Physical review letters.

[46]  A. Ozcan,et al.  Recurrent neural network-based volumetric fluorescence microscopy , 2020, Light, science & applications.

[47]  Zhou Wang,et al.  Multiscale structural similarity for image quality assessment , 2003, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.

[48]  Gabriel Popescu,et al.  Quantitative Phase Imaging , 2012 .

[49]  G. Barbastathis,et al.  Transport-of-intensity approach to differential interference contrast (TI-DIC) microscopy for quantitative phase imaging. , 2010, Optics letters.

[50]  J. Goodman.  Introduction to Fourier optics , 1969 .

[51]  Yibo Zhang,et al.  Deep learning‐based color holographic microscopy , 2019, Journal of biophotonics.

[52]  Aydogan Ozcan,et al.  Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy , 2012, Nature Methods.

[53]  Aydogan Ozcan,et al.  Field-portable wide-field microscopy of dense samples using multi-height pixel super-resolution based lensfree imaging. , 2012, Lab on a chip.

[54]  K. Nugent,et al.  Rapid quantitative phase imaging using the transport of intensity equation , 1997 .

[55]  K. Dholakia,et al.  Emergent physics-informed design of deep learning for microscopy , 2021, Journal of Physics: Photonics.

[56]  Barry R. Masters,et al.  Quantitative Phase Imaging of Cells and Tissues , 2012 .

[57]  Derek K. Tseng,et al.  Lensfree holographic imaging for on-chip cytometry and diagnostics. , 2009, Lab on a chip.

[58]  Gabriel Popescu,et al.  Quantitative phase imaging for medical diagnosis , 2017, Journal of biophotonics.

[59]  Christian Depeursinge,et al.  Quantitative phase imaging in biomedicine , 2018, Nature Photonics.

[60]  Raquel Urtasun,et al.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , 2016, NIPS.

[61]  Bahram Javidi,et al.  Extended focused image in microscopy by digital Holography. , 2005, Optics express.

[62]  Wei Luo,et al.  Propagation phasor approach for holographic image reconstruction , 2016, Scientific Reports.

[63]  Natan T Shaked,et al.  Quantitative phase microscopy of biological samples using a portable interferometer. , 2012, Optics letters.