Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning

Limited-angle tomography of strongly scattering quasi-transparent objects is a challenging, highly ill-posed problem with practical implications in medical and biological imaging, manufacturing, automation, and environmental and food security. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. Recently, it was shown that one effective way to learn the priors for strongly scattering yet highly structured 3D objects, e.g. layered and Manhattan, is by a static neural network [Goy et al, Proc. Natl. Acad. Sci. 116, 19848-19856 (2019)]. Here, we present a radically different approach where the collection of raw images from multiple angles is viewed analogously to a dynamical system driven by the object-dependent forward scattering operator. The sequence index in angle of illumination plays the role of discrete time in the dynamical system analogy. Thus, the imaging problem turns into a problem of nonlinear system identification, which also suggests dynamical learning as better fit to regularize the reconstructions. We devised a recurrent neural network (RNN) architecture with a novel split-convolutional gated recurrent unit (SC-GRU) as the fundamental building block. Through comprehensive comparison of several quantitative metrics, we show that the dynamic method improves upon previous static approaches with fewer artifacts and better overall reconstruction fidelity.

[1]  Daniel L Marks,et al.  Compressive holography of diffuse objects. , 2010, Applied optics.

[2]  Song Han,et al.  Deep Generative Adversarial Networks for Compressed Sensing Automates MRI , 2017, ArXiv.

[3]  Yunbo Wang,et al.  Eidetic 3D LSTM: A Model for Video Prediction and Beyond , 2019, ICLR.

[4]  Demetri Psaltis,et al.  Optical Tomographic Image Reconstruction Based on Beam Propagation and Sparse Regularization , 2016, IEEE Transactions on Computational Imaging.

[5]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[6]  M. Feit,et al.  Computation of mode properties in optical fiber waveguides by a propagating beam method. , 1980, Applied optics.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  A. Katsevich.  Analysis of an exact inversion algorithm for spiral cone-beam CT. , 2002, Physics in medicine and biology.

[9]  Yann LeCun,et al.  Learning Fast Approximations of Sparse Coding , 2010, ICML.

[10]  Gilbae Park,et al.  Wide viewing angle dynamic holographic stereogram with a curved array of spatial light modulators. , 2008, Optics express.

[11]  A. Devaney.  Inverse-scattering theory within the Rytov approximation. , 1981, Optics letters.

[12]  A. C. Riddle,et al.  Inversion of Fan-Beam Scans in Radio Astronomy , 1967 .

[13]  Philip S. Yu,et al.  PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning , 2018, ICML.

[14]  Michael Unser,et al.  Learning approach to optical tomography , 2015, 1502.01914.

[15]  Yuxin Hou,et al.  Multi-View Stereo by Temporal Nonparametric Fusion , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[16]  G. Wang,et al.  A general cone-beam reconstruction algorithm , 1993, IEEE Trans. Medical Imaging.

[17]  Hassan Mansour,et al.  A Recursive Born Approach to Nonlinear Inverse Scattering , 2016, IEEE Signal Processing Letters.

[18]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[19]  Gustavo K. Rohde,et al.  Optimal Mass Transport: Signal processing and machine-learning applications , 2017, IEEE Signal Processing Magazine.

[20]  Massimo Brignone,et al.  Application of the Inhomogeneous Lippmann-Schwinger Equation to Inverse Scattering Problems , 2012, SIAM J. Appl. Math..

[21]  Daniel L Marks,et al.  Compressive holography. , 2009, Optics express.

[22]  Yibo Zhang,et al.  Phase recovery and holographic image reconstruction using deep learning in neural networks , 2017, Light: Science & Applications.

[23]  Ye Duan,et al.  A multi-view recurrent neural network for 3D mesh segmentation , 2017, Comput. Graph..

[24]  M. Nikolova.  An Algorithm for Total Variation Minimization and Applications , 2004 .

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  R. A. Silverman,et al.  Wave Propagation in a Turbulent Medium , 1961 .

[27]  Jin Liu,et al.  A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-View Stereo Reconstruction From an Open Aerial Dataset , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  I. Daubechies,et al.  An iterative thresholding algorithm for linear inverse problems with a sparsity constraint , 2003, math/0307152.

[29]  P. Grangeat.  Mathematical framework of cone beam 3D reconstruction via the first derivative of the radon transform , 1991 .

[30]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[31]  J. Radon.  On the determination of functions from their integral values along certain manifolds , 1986, IEEE Transactions on Medical Imaging.

[32]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[33]  Marc Teboulle,et al.  Fast Gradient-Based Algorithms for Constrained Total Variation Image Denoising and Deblurring Problems , 2009, IEEE Transactions on Image Processing.

[34]  Michael Unser,et al.  Optical Tomography based on a nonlinear model that handles multiple scattering , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[35]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Guofan Jin,et al.  Twin-Image-Free Holography: A Compressive Sensing Approach. , 2018, Physical review letters.

[37]  Z Q Lu,et al.  JKM Perturbation Theory, Relaxation Perturbation Theory, and Their Applications to Inverse Scattering: Theory and Reconstruction Algorithms. , 1986, IEEE transactions on ultrasonics, ferroelectrics, and frequency control.

[38]  D. Donoho,et al.  Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery , 2017, ArXiv.

[39]  Philip S. Yu,et al.  PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs , 2017, NIPS.

[40]  Yu Zhang,et al.  Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM , 2017, INTERSPEECH.

[41]  Morteza Mardani,et al.  Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery , 2017, ArXiv.

[42]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[43]  Michael Unser,et al.  Learning Tomography Assessed Using Mie Theory , 2018 .

[44]  Laura Waller,et al.  High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images. , 2019, Optica.

[45]  Jerome H Milgram,et al.  Computational reconstruction of images from holograms. , 2002, Applied optics.

[46]  Hiroyuki Kudo,et al.  Helical-scan computed tomography using cone-beam projections , 1991, Conference Record of the 1991 IEEE Nuclear Science Symposium and Medical Imaging Conference.

[47]  Benjamin Schrauwen,et al.  An overview of reservoir computing: theory, applications and implementations , 2007, ESANN.

[48]  Michael Unser,et al.  Deep Convolutional Neural Network for Inverse Problems in Imaging , 2016, IEEE Transactions on Image Processing.

[49]  P. Dreike,et al.  Convolution reconstruction of fan beam projections , 1976 .

[50]  George Barbastathis,et al.  Imaging through glass diffusers using densely connected convolutional networks , 2017, Optica.

[51]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[52]  Daniel L Marks,et al.  A family of approximations spanning the Born and Rytov scattering series. , 2006, Optics express.

[53]  Fathi M. Salem,et al.  Gate-variants of Gated Recurrent Unit (GRU) neural networks , 2017, 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS).

[54]  Chao Yang,et al.  Shape Inpainting Using 3D Generative Adversarial Network and Recurrent Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[55]  A. Ozcan,et al.  Deep learning in holography and coherent imaging , 2019, Light: Science & Applications.

[56]  A. Ozcan,et al.  On the use of deep learning for computational imaging , 2019, Optica.

[57]  Mario Bertero,et al.  Introduction to Inverse Problems in Imaging , 1998 .

[58]  Z Q Lu,et al.  Multidimensional structure diffraction tomography for varying object orientation through generalised scattered waves , 1985 .

[59]  W. Chew,et al.  Reconstruction of two-dimensional permittivity distribution using the distorted Born iterative method. , 1990, IEEE transactions on medical imaging.

[60]  Aydogan Ozcan,et al.  Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram , 2018, Light: Science & Applications.

[61]  Ruqiang Yan,et al.  Learning to Monitor Machine Health with Convolutional Bi-Directional LSTM Networks , 2017, Sensors.

[62]  C. Fang-Yen,et al.  Tomographic phase microscopy , 2008, Nature Methods.

[63]  G. Nehmetallah,et al.  Digital tomographic compressive holographic reconstruction of three-dimensional objects in transmissive and reflective geometries. , 2013, Applied optics.

[64]  Anthony J. Devaney,et al.  Higher order (nonlinear) diffraction tomography: Inversion of the Rytov series , 2000, IEEE Trans. Inf. Theory.

[65]  Yann Dauphin,et al.  A Convolutional Encoder Model for Neural Machine Translation , 2016, ACL.

[66]  George Barbastathis,et al.  High-resolution limited-angle phase tomography of dense layered objects using deep neural networks , 2018, Proceedings of the National Academy of Sciences.

[67]  Benjamin Schrauwen,et al.  Reservoir Computing Trends , 2012, KI - Künstliche Intelligenz.

[68]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[69]  George Barbastathis,et al.  Low Photon Count Phase Retrieval Using Deep Learning. , 2018, Physical review letters.

[70]  Michael Unser,et al.  Three-Dimensional Optical Diffraction Tomography With Lippmann-Schwinger Model , 2020, IEEE Transactions on Computational Imaging.

[71]  Jürgen Schmidhuber,et al.  Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation , 2015, NIPS.

[72]  L. Tian,et al.  Quantitative measurement of size and three-dimensional position of fast-moving bubbles in air-water mixture flows using digital holography. , 2010, Applied optics.

[73]  Emmanuel J. Candès,et al.  Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information , 2004, IEEE Transactions on Information Theory.

[74]  Byoungho Lee,et al.  Recent progress in three-dimensional information processing based on integral imaging. , 2009, Applied optics.

[75]  Shuai Li,et al.  Lensless computational imaging through deep learning , 2017, ArXiv.

[76]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[77]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[78]  Ulugbek Kamilov,et al.  Efficient and accurate inversion of multiple scattering with deep learning , 2018, Optics express.

[79]  E. Wolf.  Three-dimensional structure determination of semi-transparent objects from holographic data , 1969 .

[80]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[81]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[82]  L. Feldkamp,et al.  Practical cone-beam algorithm , 1984 .

[83]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[84]  G. Nehmetallah,et al.  Applications of digital and analog holography in three-dimensional imaging , 2012 .

[85]  C. Villani.  Topics in Optimal Transportation , 2003 .

[86]  Suchendra M. Bhandarkar,et al.  DepthNet: A Recurrent Neural Network Architecture for Monocular Depth Prediction , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).