3D Scattering Tomography by Deep Learning with Architecture Tailored to Cloud Fields

We present 3DeepCT, a deep neural network for computed tomography, which performs 3D reconstruction of scattering volumes from multi-view images. Our architecture is dictated by the stationary nature of atmospheric cloud fields. The task of volumetric scattering tomography aims at recovering a volume from its 2D projections. This problem has been studied extensively, leading, to diverse inverse methods based on signal processing and physics models. However, such techniques are typically iterative, exhibiting high computational load and long convergence time. We show that 3DeepCT outperforms physics-based inverse scattering methods in term of accuracy as well as offering a significant orders of magnitude improvement in computational time. To further improve the recovery accuracy, we introduce a hybrid model that combines 3DeepCT and physics-based method. The resultant hybrid technique enjoys fast inference time and improved recovery performance.

[1]  Simon Arridge,et al.  Quantitative photoacoustic tomography using forward and adjoint Monte Carlo models of radiance , 2016, Journal of biomedical optics.

[2]  Frits H. Post,et al.  A statistical approach to the life cycle analysis of cumulus clouds selected in a virtual reality environment , 2009 .

[3]  Pingkun Yan,et al.  Sensorless Freehand 3D Ultrasound Reconstruction via Deep Contextual Learning , 2020, MICCAI.

[4]  Yoav Y. Schechner,et al.  The Next Best Underwater View , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Honglak Lee,et al.  Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision , 2016, NIPS.

[6]  M. Zelinka,et al.  Cloud feedback mechanisms and their representation in global climate models , 2017 .

[7]  Gustavo Carneiro,et al.  Scaling CNNs for High Resolution Volumetric Reconstruction from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[8]  Rajiv Gupta,et al.  X-Ray Computed Tomography Through Scatter , 2018, ECCV.

[9]  Yoav Y. Schechner,et al.  In-situ multi-view multi-scattering stochastic tomography , 2016, 2016 IEEE International Conference on Computational Photography (ICCP).

[10]  A. Pokrovsky,et al.  Simulation of effects of atmospheric aerosols on deep turbulent convective clouds using a spectral microphysics mixed-phase cumulus cloud model. Part I: Model description and possible applications , 2004 .

[11]  Anthony B. Davis,et al.  Airborne Three-Dimensional Cloud Tomography , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[12]  Yoav Y. Schechner,et al.  Clouds in the Cloud , 2014, ACCV.

[13]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[14]  Roel Neggers,et al.  Size Statistics of Cumulus Cloud Populations in Large-Eddy Simulations , 2003 .

[15]  B. Mayer.  Radiative transfer in the cloudy atmosphere , 2009 .

[16]  Shengping Zhang,et al.  Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[17]  Feng Zhang,et al.  CloudNet: Ground‐Based Cloud Classification With Deep Convolutional Neural Network , 2018, Geophysical Research Letters.

[18]  Jan Kleissl,et al.  Cloud tomography applied to sky images: A virtual testbed , 2018, Solar Energy.

[19]  Jun Chen,et al.  GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[20]  Kai Yang,et al.  Signal Sensing and Reconstruction Paradigms for a Novel Multi-Source Static Computed Tomography System , 2020, ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[21]  Jonathan T. Barron,et al.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis , 2020, ECCV.

[22]  She Kun,et al.  A Fast Single-Image Dehazing Algorithm Based on Dark Channel Prior and Rayleigh Scattering , 2020, IEEE Access.

[23]  Henrique M. J. Barbosa,et al.  The Harp Hype Ran Gular Imaging Polarimeter and the Need for Small Satellite Payloads with High Science Payoff for Earth Science Remote Sensing , 2018, IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium.

[24]  Quanzheng Li,et al.  Computationally Efficient Cascaded Training for Deep Unrolled Network in CT Imaging , 2018, Medical physics.

[25]  Marcel van Gerven,et al.  Deep disentangled representations for volumetric reconstruction , 2016, ECCV Workshops.

[26]  Shree K. Nayar,et al.  Multiplexing for Optimal Lighting , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Toshiya Mori,et al.  Computed tomography reconstruction of SO2 concentration distribution in the volcanic plume of Miyakejima, Japan, by airborne traverse technique using three UV spectrometers , 2008 .

[28]  Yoav Y. Schechner,et al.  Distributed Sky Imaging Radiometry and Tomography , 2020, 2020 IEEE International Conference on Computational Photography (ICCP).

[29]  Navalgund Rao,et al.  Deep 3D convolution neural network for CT brain hemorrhage classification , 2018, Medical Imaging.

[30]  Wolfgang Osten,et al.  Exploiting scattering media for exploring 3D objects , 2016, Light: Science & Applications.

[31]  Michael Elad,et al.  Spatially-Adaptive Reconstruction in Computed Tomography Using Neural Networks , 2013, IEEE Transactions on Medical Imaging.

[32]  G. Rybicki.  Radiative transfer , 2019, Climate Change and Terrestrial Ecosystem Modeling.

[33]  A. P. Siebesma,et al.  A Large Eddy Simulation Intercomparison Study of Shallow Cumulus Convection , 2003 .

[34]  Steve Marschner,et al.  Matching Real Fabrics with Micro-Appearance Models , 2015, ACM Trans. Graph..

[35]  David Iluz,et al.  What is the Space of Attenuation Coefficients in Underwater Computer Vision? , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Shuang Zhao,et al.  Inverse volume rendering with material dictionaries , 2013, ACM Trans. Graph..

[37]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Sebastian Nowozin,et al.  Occupancy Networks: Learning 3D Reconstruction in Function Space , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Anat Levin,et al.  An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering , 2016, ECCV.

[40]  Laura Waller,et al.  A multiple scattering algorithm for three dimensional phase contrast atomic electron tomography. , 2019, Ultramicroscopy.

[41]  Qian Liu,et al.  Hyperspectral Infrared Sounder Cloud Detection Using Deep Neural Network Model , 2022, IEEE Geoscience and Remote Sensing Letters.

[42]  R. Raskar,et al.  All Photons Imaging Through Volumetric Scattering , 2016, Scientific Reports.

[43]  Wei Zhao,et al.  Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning , 2019, Nature Biomedical Engineering.

[44]  M. King,et al.  Determination of the Optical Thickness and Effective Particle Radius of Clouds from Reflected Solar Radiation Measurements. Part II: Marine Stratocumulus Observations , 1991 .

[45]  Jiapan Guo,et al.  Automatic segmentation of the mandible from computed tomography scans for 3D virtual surgical planning using the convolutional neural network , 2019, Physics in medicine and biology.

[46]  Anthony B. Davis,et al.  Multi-view polarimetric scattering cloud tomography and retrieval of droplet size , 2020, Remote. Sens..

[47]  Kavita Bala,et al.  Towards Learning-based Inverse Subsurface Scattering , 2020, 2020 IEEE International Conference on Computational Photography (ICCP).

[48]  A. P. Siebesma,et al.  Clouds, circulation and climate sensitivity , 2015 .

[49]  Alexander Khain,et al.  Ice Formation in Arctic Mixed-Phase Clouds: Insights from a 3-D Cloud-Resolving Model with Size-Resolved Aerosol and Cloud Microphysics , 2009 .

[50]  Hao Zhang,et al.  Learning Implicit Fields for Generative Shape Modeling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[52]  Kavita Bala,et al.  Langevin monte carlo rendering with gradient-based adaptation , 2020, ACM Trans. Graph..

[53]  Yoav Y. Schechner,et al.  Monotonicity Prior for Cloud Tomography , 2020, ECCV.

[54]  Robert Pincus,et al.  Computational Cost and Accuracy in Calculating Three-Dimensional Radiative Transfer: Results for New Implementations of Monte Carlo and SHDOM , 2009 .

[55]  Duygu Ceylan,et al.  DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction , 2019, NeurIPS.

[56]  Céline Cornet,et al.  Three-dimensional polarized Monte Carlo atmospheric radiative transfer model (3DMCPOL): 3D effects on polarized visible reflectances of a cirrus cloud , 2010 .

[57]  Claudia Emde,et al.  A 3-D polarized reversed Monte Carlo radiative transfer model for Millimeter and submillimeter passive remote sensing in cloudy atmospheres , 2005, IEEE Transactions on Geoscience and Remote Sensing.

[58]  D. Murtagh,et al.  The MATS satellite mission – gravity wave studies by Mesospheric Airglow/Aerosol Tomography and Spectroscopy , 2020 .

[59]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[60]  Ko Nishino,et al.  Factorizing Scene Albedo and Depth from a Single Foggy Image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[61]  José Baruchel,et al.  X-Ray Tomography in Material Science , 2000 .

[62]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[63]  K. Evans.  The Spherical Harmonics Discrete Ordinate Method for Three-Dimensional Atmospheric Radiative Transfer , 1998 .

[64]  R. Grainger,et al.  Calculation of Mie derivatives. , 2004, Applied optics.

[65]  D. Randall,et al.  Cloud resolving modeling of the ARM summer 1997 IOP: Model formulation, results, uncertainties, and sensitivities , 2003 .

[66]  Anthony B. Davis,et al.  Multiple-Scattering Microphysics Tomography , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[67]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[68]  M. King,et al.  Determination of the optical thickness and effective particle radius of clouds from reflected solar , 1990 .

[69]  G. Feingold,et al.  Large-Eddy Simulations of Trade Wind Cumuli: Investigation of Aerosol Indirect Effects , 2006 .

[70]  D. Efremenko,et al.  A multi-dimensional vector spherical harmonics discrete ordinate method for atmospheric radiative transfer , 2013 .

[71]  Abhinav Gupta,et al.  Learning a Predictable and Generative Vector Representation for Objects , 2016, ECCV.