DeepRegularizer: Rapid Resolution Enhancement of Tomographic Imaging Using Deep Learning

Optical diffraction tomography measures the three-dimensional refractive index map of a specimen and visualizes biochemical phenomena at the nanoscale in a non-destructive manner. One major drawback of optical diffraction tomography is poor axial resolution due to limited access to the three-dimensional optical transfer function. This missing cone problem has been addressed through regularization algorithms that use a priori information, such as non-negativity and sample smoothness. However, the iterative nature of these algorithms and their parameter dependency make real-time visualization impossible. In this article, we propose and experimentally demonstrate a deep neural network, which we term DeepRegularizer, that rapidly improves the resolution of a three-dimensional refractive index map. Trained with pairs of datasets (a raw refractive index tomogram and a resolution-enhanced refractive index tomogram via the iterative total variation algorithm), the three-dimensional U-net-based convolutional neural network learns a transformation between the two tomogram domains. The feasibility and generalizability of our network are demonstrated using bacterial cells and a human leukaemic cell line, and by validating the model across different samples. DeepRegularizer offers more than an order of magnitude faster regularization performance compared to the conventional iterative method. We envision that the proposed data-driven approach can bypass the high time complexity of various image reconstructions in other imaging modalities.

[1]  Jochen Guck,et al.  Single-cell diffraction tomography with optofluidic rotation about a tilted axis , 2015, SPIE NanoScience + Engineering.

[2]  Vittorio Bianco,et al.  How holographic imaging can improve machine learning , 2019, Optical Metrology.

[3]  Hyun-seok Min,et al.  Quantitative Phase Imaging and Artificial Intelligence: A Review , 2018, IEEE Journal of Selected Topics in Quantum Electronics.

[4]  YongKeun Park,et al.  Refractive index tomograms and dynamic membrane fluctuations of red blood cells from patients with diabetes mellitus , 2016, Scientific Reports.

[5]  Yibo Zhang,et al.  PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning , 2018, Light: Science & Applications.

[6]  Piotr Makowski,et al.  Generalized total variation iterative constraint strategy in limited angle optical diffraction tomography. , 2016, Optics express.

[7]  Yoram Bresler,et al.  Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography , 1998, IEEE Trans. Image Process..

[8]  Fei Wang,et al.  Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging. , 2019, Optics express.

[9]  Bin Dong,et al.  Stitching sub-aperture in digital holography based on machine learning. , 2020, Optics express.

[10]  D. Donoho,et al.  Sparse MRI: The application of compressed sensing for rapid MR imaging , 2007, Magnetic resonance in medicine.

[11]  George Barbastathis,et al.  High-resolution limited-angle phase tomography of dense layered objects using deep neural networks , 2018, Proceedings of the National Academy of Sciences.

[12]  Frederic Noo,et al.  A local shift-variant Fourier model and experimental validation of circular cone-beam computed tomography artifacts. , 2009, Medical physics.

[13]  K. Tam,et al.  Tomographical imaging with limited-angle input , 1981 .

[14]  Jin Won Kim,et al.  Label-Free Tomographic Imaging of Lipid Droplets in Foam Cells for Machine-Learning-Assisted Therapeutic Evaluation of Targeted Nanodrugs. , 2020, ACS nano.

[15]  Tomer Michaeli,et al.  Deep-STORM: super-resolution single-molecule microscopy by deep learning , 2018, 1801.09631.

[16]  V. Lauer.  New approach to optical diffraction tomography yielding a vector equation of diffraction tomography and a novel tomographic microscope , 2002, Journal of microscopy.

[17]  Lei Tian,et al.  Deep learning approach for Fourier ptychography microscopy. , 2018, Optics express.

[18]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[19]  Xiao Huang,et al.  Fiber bundle image restoration using deep learning. , 2019, Optics letters.

[20]  Aydogan Ozcan,et al.  Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning , 2019, Nature Methods.

[21]  Hao Wang,et al.  Exploit imaging through opaque wall via deep learning , 2017, ArXiv.

[22]  Yongjin Sung,et al.  Stain-Free Quantification of Chromosomes in Live Cells Using Regularized Tomographic Phase Microscopy , 2012, PloS one.

[23]  Yongjin Sung,et al.  Deterministic regularization of three-dimensional optical diffraction tomography. , 2011, Journal of the Optical Society of America. A, Optics, image science, and vision.

[24]  Laura Waller,et al.  High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images. , 2019, Optica.

[25]  Seungwoo Shin,et al.  Generalized quantification of three-dimensional resolution in optical diffraction tomography using the projection of maximal spatial bandwidths. , 2018, Journal of the Optical Society of America. A, Optics, image science, and vision.

[26]  Andrea Vedaldi,et al.  Deep Image Prior , 2017, International Journal of Computer Vision.

[27]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[28]  Kees Joost Batenburg,et al.  Electron tomography based on a total variation minimization reconstruction technique , 2012 .

[29]  Taesup Kim,et al.  Scalable Neural Architecture Search for 3D Medical Image Segmentation , 2019, MICCAI.

[30]  C. Depeursinge,et al.  Quantitative phase imaging in biomedicine , 2012, 2012 Conference on Lasers and Electro-Optics (CLEO).

[31]  Demetri Psaltis,et al.  High-fidelity optical diffraction tomography of multiple scattering samples , 2019, Light: Science & Applications.

[32]  Zhuo Wang,et al.  Spatial light interference tomography (SLIT) , 2011, Optics express.

[33]  A. Ozcan,et al.  Deep learning in holography and coherent imaging , 2019, Light: Science & Applications.

[34]  M. Schmid.  Principles Of Optics Electromagnetic Theory Of Propagation Interference And Diffraction Of Light , 2016 .

[35]  Gunho Choi,et al.  Cycle-consistent deep learning approach to coherent noise reduction in optical diffraction tomography. , 2018, Optics express.

[36]  George Barbastathis,et al.  Imaging through glass diffusers using densely connected convolutional networks , 2017, Optica.

[37]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[38]  Wei Chen,et al.  Optical Radiomic Signatures Derived from Optical Coherence Tomography Images Improve Identification of Melanoma. , 2019, Cancer research.

[39]  Christophe Zimmer,et al.  Deep learning massively accelerates super-resolution localization microscopy , 2018, Nature Biotechnology.

[40]  Kevin C Zhou,et al.  Diffraction tomography with a deep image prior. , 2019, Optics express.

[41]  Sandeep Subramanian,et al.  Deep Complex Networks , 2017, ICLR.

[42]  Young Jae Lee,et al.  PICS: Phase Imaging with Computational Specificity , 2020 .

[43]  Michael Unser,et al.  Three-Dimensional Optical Diffraction Tomography With Lippmann-Schwinger Model , 2020, IEEE Transactions on Computational Imaging.

[44]  C. Fang-Yen,et al.  Tomographic phase microscopy , 2008, Nature Methods.

[45]  Samuel J. Yang,et al.  In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images , 2018, Cell.

[46]  E. Wolf.  Three-dimensional structure determination of semi-transparent objects from holographic data , 1969 .

[47]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[48]  E. Cuche,et al.  Cell refractive index tomography by digital holographic microscopy. , 2006, Optics letters.

[49]  Pasquale Memmolo,et al.  Tomographic flow cytometry by digital holography , 2016, Light: Science & Applications.

[50]  YoungJu Jo,et al.  Quantitative Phase Imaging Techniques for the Study of Cell Pathophysiology: From Principles to Applications , 2013, Sensors.

[51]  Demetri Psaltis,et al.  Optical Tomographic Image Reconstruction Based on Beam Propagation and Sparse Regularization , 2016, IEEE Transactions on Computational Imaging.

[52]  Ji-Ho Park,et al.  Label-free high-resolution 3-D imaging of gold nanoparticles inside live cells using optical diffraction tomography , 2016, bioRxiv.

[53]  Jong Chul Ye,et al.  Comparative study of iterative reconstruction algorithms for missing cone problems in optical diffraction tomography. , 2015, Optics express.

[54]  Tae-Jin Je,et al.  Enhancement of optical resolution in three-dimensional refractive-index tomograms of biological samples by employing micromirror-embedded coverslips. , 2018, Lab on a chip.

[55]  Tom Goldstein,et al.  The Split Bregman Method for L1-Regularized Problems , 2009, SIAM J. Imaging Sci..

[56]  Thomas Neumann,et al.  Three-dimensional imaging of single isolated cell nuclei using optical projection tomography. , 2005, Optics express.

[57]  Demetri Psaltis,et al.  Multimode optical fiber transmission with a deep learning network , 2018, Light: Science & Applications.

[58]  Gabriel Popescu,et al.  Phase imaging with computational specificity (PICS) for measuring dry mass changes in sub-cellular compartments , 2020, Nature Communications.

[59]  Rayyan Manwar,et al.  Deep learning protocol for improved photoacoustic brain imaging , 2020, Journal of biophotonics.

[60]  YongKeun Park,et al.  Label-free non-invasive quantitative measurement of lipid contents in individual microalgal cells using refractive index tomography , 2017, Scientific Reports.

[61]  A. Ozcan,et al.  On the use of deep learning for computational imaging , 2019, Optica.

[62]  E. Cuche,et al.  Spatial filtering for zero-order and twin-image elimination in digital off-axis holography. , 2000, Applied optics.

[63]  Wonseok Jeon,et al.  Speckle noise reduction for digital holographic images using multi-scale convolutional neural networks. , 2018, Optics letters.