Deep Volumetric Ambient Occlusion

We present a novel deep learning based technique for volumetric ambient occlusion in the context of direct volume rendering. Our proposed Deep Volumetric Ambient Occlusion (DVAO) approach can predict per-voxel ambient occlusion in volumetric data sets, while considering global information provided through the transfer function. The proposed neural network only needs to be executed upon change of this global information, and thus supports real-time volume interaction. Accordingly, we demonstrate DVAO's ability to predict volumetric ambient occlusion, such that it can be applied interactively within direct volume rendering. To achieve the best possible results, we propose and analyze a variety of transfer function representations and injection strategies for deep neural networks. Based on the obtained results we also give recommendations applicable in similar volume learning scenarios. Lastly, we show that DVAO generalizes to a variety of modalities, despite being trained on computed tomography data only.

[1]  Timo Ropinski,et al.  About the Influence of Illumination Models on Image Comprehension in Direct Volume Rendering , 2011, IEEE Transactions on Visualization and Computer Graphics.

[2]  Daniel Weiskopf,et al.  Ambient Volume Scattering , 2013, IEEE Transactions on Visualization and Computer Graphics.

[3]  Timo Ropinski,et al.  Inviwo — A Visualization System with Usage Abstraction Levels , 2018, IEEE Transactions on Visualization and Computer Graphics.

[4]  Mathias Schott,et al.  A Directional Occlusion Shading Model for Interactive Direct Volume Rendering , 2009, Comput. Graph. Forum.

[5]  Thomas Ertl,et al.  Local Prediction Models for Spatiotemporal Volume Visualization , 2019, IEEE Transactions on Visualization and Computer Graphics.

[6]  Germain Forestier,et al.  Deep learning for time series classification: a review , 2018, Data Mining and Knowledge Discovery.

[7]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Timo Aila,et al.  Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder , 2017, ACM Trans. Graph..

[9]  Liyuan Liu,et al.  On the Variance of the Adaptive Learning Rate and Beyond , 2019, ICLR.

[10]  Stefan Bruckner,et al.  Interactive Dynamic Volume Illumination with Refraction and Caustics , 2018, IEEE Transactions on Visualization and Computer Graphics.

[11]  Rüdiger Westermann,et al.  Volumetric Isosurface Rendering with Deep Learning-Based Super-Resolution , 2019, IEEE Transactions on Visualization and Computer Graphics.

[12]  Pere-Pau Vázquez,et al.  Real-time ambient occlusion and halos with Summed Area Tables , 2010, Comput. Graph..

[13]  Hyun-Chul Kim,et al.  3D convolutional neural network for feature extraction and classification of fMRI volumes , 2018, 2018 International Workshop on Pattern Recognition in Neuroimaging (PRNI).

[14]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[15]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[16]  Ivan Viola,et al.  A Multidirectional Occlusion Shading Model for Direct Volume Rendering , 2010, Comput. Graph. Forum.

[17]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[18]  Hao Wu,et al.  Mixed Precision Training , 2017, ICLR.

[19]  Elmar Eisemann,et al.  Smooth Probabilistic Ambient Occlusion for Volume Rendering , 2018, GPU Pro 360.

[20]  Xiaoru Yuan,et al.  DNN-VolVis: Interactive Volume Visualization Supported by Deep Neural Network , 2019, 2019 IEEE Pacific Visualization Symposium (PacificVis).

[21]  Timo Ropinski,et al.  Interactive Volume Rendering with Dynamic Ambient Occlusion and Color Bleeding , 2008, Comput. Graph. Forum.

[22]  Hans-Peter Seidel,et al.  Deep Shading: Convolutional Neural Networks for Screen Space Shading , 2016, Comput. Graph. Forum.

[23]  Anders Ynnerman,et al.  Local Ambient Occlusion in Direct Volume Rendering , 2010, IEEE Transactions on Visualization and Computer Graphics.

[24]  Mathias Schott,et al.  Ambient Occlusion Effects for Combined Volumes and Tubular Geometry , 2013, IEEE Transactions on Visualization and Computer Graphics.

[25]  Aaron Knoll,et al.  OSPRay - A CPU Ray Tracing Framework for Scientific Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[26]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Diganta Misra,et al.  Mish: A Self Regularized Non-Monotonic Neural Activation Function , 2019, ArXiv.

[28]  Alan C. Evans,et al.  BrainWeb: Online Interface to a 3D MRI Simulated Brain Database , 1997 .

[29]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[30]  Jens H. Krüger,et al.  State of the Art in Transfer Functions for Direct Volume Rendering , 2016, Comput. Graph. Forum.

[31]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[32]  Mert R. Sabuncu,et al.  3D Convolutional Neural Networks for Classification of Functional Connectomes , 2018, DLMIA/ML-CDS@MICCAI.

[33]  Carsten Dachsbacher,et al.  Anisotropic Ambient Volume Shading , 2016, IEEE Transactions on Visualization and Computer Graphics.

[34]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Nelson L. Max,et al.  Optical Models for Direct Volume Rendering , 1995, IEEE Trans. Vis. Comput. Graph..

[36]  Joshua A. Levine,et al.  A Generative Model for Volume Rendering , 2017, IEEE Transactions on Visualization and Computer Graphics.

[37]  Won-Ki Jeong,et al.  An Intelligent System Approach for Probabilistic Volume Rendering Using Hierarchical 3D Convolutional Sparse Coding , 2018, IEEE Transactions on Visualization and Computer Graphics.

[38]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[39]  Eric Krokos,et al.  Deep-Learning-Assisted Volume Visualization , 2019, IEEE Transactions on Visualization and Computer Graphics.

[40]  Charl P. Botha,et al.  Exposure Render: An Interactive Photo-Realistic Volume Rendering Framework , 2012, PloS one.

[41]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[42]  Timo Aila,et al.  A Style-Based Generator Architecture for Generative Adversarial Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Geoffrey E. Hinton,et al.  Lookahead Optimizer: k steps forward, 1 step back , 2019, NeurIPS.

[44]  Navalgund Rao,et al.  Deep 3D convolution neural network for CT brain hemorrhage classification , 2018, Medical Imaging.

[45]  Anders Ynnerman,et al.  Correlated Photon Mapping for Interactive Global Illumination of Time-Varying Volumetric Data , 2017, IEEE Transactions on Visualization and Computer Graphics.

[46]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[47]  Rohit Ghosh,et al.  Development and Validation of Deep Learning Algorithms for Detection of Critical Findings in Head CT Scans , 2018, ArXiv.

[48]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.