Zero-Shot Learning of Continuous 3D Refractive Index Maps from Discrete Intensity-Only Measurements

Intensity diffraction tomography (IDT) refers to a class of optical microscopy techniques for imaging the 3D refractive index (RI) distribution of a sample from a set of 2D intensity-only measurements. The reconstruction of artifact-free RI maps is a fundamental challenge in IDT due to the loss of phase information and the missing cone problem. Neural fields (NF) has recently emerged as a new deep learning (DL) paradigm for learning continuous representations of complex 3D scenes without external training datasets. We present DeCAF as the first NF-based IDT method that can learn a high-quality continuous representation of a RI volume directly from its intensity-only and limited-angle measurements. We show on three different IDT modalities and multiple biological samples that DeCAF can generate high-contrast and artifact-free RI maps. Refractive index (RI) is an optical property that describes the interaction between light and matter within a sample. The real part of RI characterizes the phase velocity while its imaginary part characterizes the attenuation. RI can thus serve as an endogenous source of optical contrast for imaging samples without staining or labeling them using external contrast agents. By quantitatively characterizing the three-dimensional (3D) distribution of the RI, one can visualize cellular or subcellular structures useful in morphogenesis [1], oncology [2], cellular pathophysiology [3], biochemistry [4], and beyond (see the review papers [5,6]). Intensity diffraction tomography (IDT) is a recent technique for producing 3D RI maps of a sample by measuring the light it diffracts. In the standard IDT setup, a sample is illuminated multiple times from different angles, and a set of two-dimensional (2D) intensity projections are captured by the camera (see Figure 1(a)). A tomographic image reconstruction algorithm is then used to computationally reconstruct the desired 3D RI distribution from the set of 2D measurements. Unlike traditional optical diffraction tomography (ODT) that uses interferometry to record the complex-valued light fields [7–9], IDT only measures the squared amplitude of the scattered light, leading to an easy setup on standard transmission optical microscopes with inexpensive hardware modifications. Such flexibility has spurred different IDT variants integrating object scanning [10,11], angled illumination [12–15], pupil engineering [16,17], and multiple scattering [18,19]. Setups achieving high resolution [18] and fast acquisition [20] have also been reported. 1 ar X iv :2 11 2. 00 00 2v 1 [ ee ss .I V ] 2 7 N ov 2 02 1 Despite the rich literature on IDT, image reconstruction remains a fundamental challenge. The first issue is that the phase of the scattered light field is missing from the measurements, resulting in a nonlinear measurement system that is not characterizable by the classical linear Fourier diffraction theory [21]. This rules out the usage of the standard filtered-backprojection methods and calls for advanced computational algorithms. The second issue is the well-known missing cone problem, causing elongation of the object along the optical axis (z dimension) and hence reduction of the axial resolution. The missing cone problem is due to the limited-angle tomographic setup, where illuminations can come only from one side of the sample plane with a limited range for angle variation (less than about 40◦ in our setups). This leads to an incomplete coverage of the 3D Fourier spectra with a cone-shape missing region in the axial direction. These missing phase and missing cone problems make image reconstruction in IDT a severely ill-posed inverse problem. Regularization methods are widely-used for mitigating ill-posed nature of many inverse problems. These methods are based on minimizing a cost function consisting of a data-fidelity term and a regularization term, where the former uses a physical-model to quantify the mismatch between the predicted and acquired measurements, while the latter promotes solutions that are consistent with a priori knowledge on the sample. For example, the least-squares loss and Tikhonov regularizer (`2penalty) are widely-used for obtaining closed-form solutions to inverse problems [14]. The work on plug-and-play priors has generalized the notion of image priors to implicit regularizers characterized by image denoisers [22–25]. Recently, deep learning (DL) has emerged as a powerful paradigm for image reconstruction. A traditional DL reconstruction is based on training a convolutional neural network (CNN) over a large dataset to learn a mapping from low-quality images to their high-quality counterparts. The state-of-the-art performance of such methods has been demonstrated in X-ray computed tomography [26,27], magnetic resonance imaging [28,29], optical tomography [30,31], and seismic imaging [32] (see the reviews [33–35]). While DL has significantly improved image reconstruction in many modalities, traditional DL methods are impractical for image reconstruction in IDT where it is difficult to acquire high-quality ground-truth RI maps in experiments. Although a physics-based simulator has been proposed to generate datasets for training IDT artifact-suppressing CNNs, the results are still limited by the mismatch between the simulation and experiments [36]. Neural fields (NF) is a recent DL paradigm that has gained popularity in computer vision and graphics for representing and rendering 3D scenes using coordinate-based deep neural networks [37, 38]. It is worth mentioning that while NF was suggested to be the most appropriate term [39,40], this idea currently goes by various names in the vision/graphics literature, including neural coordinatebased representations or neural implicit models. It has been shown that NF can learn a high-quality representation of a complex scene from a sparse set of data without any external training dataset. Motivated by this property, we propose Deep Continuous Artifact-free RI Field (DeCAF) as a first NF-based IDT method for learning a high-quality continuous 3D RI map from intensity-only and limited-angle measurements without any external training dataset of ground-truth RI maps. Figure 1 provides a conceptual illustration of DeCAF. The key features of DeCAF are as follows: • The central component of DeCAF is a multilayer perceptron (MLP), which is a fully-connected (non-convolutional) deep network, for learning a function that maps 3D coordinates (x, y, z) to the corresponding RI values. The trained MLP thus provides a continuous neural representation of the RI map. The RI value at any spatial location can be retrieved by querying the trained MLP with the corresponding coordinate. By decoupling representation from an explicit voxel grid, DeCAF can efficiently store and process large 3D volumes. • DeCAF is a zero-shot method, meaning that it does not require training using an external 2