A Generative Model for Volume Rendering

We present a technique to synthesize and analyze volume-rendered images using generative models. We use the Generative Adversarial Network (GAN) framework to compute a model from a large collection of volume renderings, conditioned on (1) viewpoint and (2) transfer functions for opacity and color. Our approach facilitates tasks for volume analysis that are challenging to achieve using existing rendering techniques such as ray casting or texture-based methods. We show how to guide the user in transfer function editing by quantifying expected change in the output image. Additionally, the generative model transforms transfer functions into a view-invariant latent space specifically designed to synthesize volume-rendered images. We use this space directly for rendering, enabling the user to explore the space of volume-rendered images. As our model is independent of the choice of volume rendering process, we show how to analyze volume-rendered images produced by direct and global illumination lighting, for a variety of volume datasets.

[1]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[2]  Enrico Gobbetti,et al.  COVRA: A compression‐domain output‐sensitive volume rendering architecture based on a sparse representation of voxel blocks , 2012, Comput. Graph. Forum.

[3]  Christof Rezk-Salama,et al.  High-Level User Interfaces for Transfer Function Design with Semantics , 2006, IEEE Transactions on Visualization and Computer Graphics.

[4]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[5]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[6]  Carla Maria Dal Sasso Freitas,et al.  Design of Multi-dimensional Transfer Functions Using Dimensional Reduction , 2007, EuroVis.

[7]  Thomas Schultz,et al.  Learning Probabilistic Transfer Functions: A Comparative Study of Classifiers , 2015, Comput. Graph. Forum.

[8]  Jens H. Krüger,et al.  State of the Art in Transfer Functions for Direct Volume Rendering , 2016, Comput. Graph. Forum.

[9]  Stefan Bruckner,et al.  Eurographics/ Ieee-vgtc Symposium on Visualization 2010 Isosurface Similarity Maps , 2022 .

[10]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[11]  Abhinav Gupta,et al.  Generative Image Modeling Using Style and Structure Adversarial Networks , 2016, ECCV.

[12]  Kwan-Liu Ma,et al.  Fuzzy Volume Rendering , 2012, IEEE Transactions on Visualization and Computer Graphics.

[13]  James P. Ahrens,et al.  An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis , 2014, SC14: International Conference for High Performance Computing, Networking, Storage and Analysis.

[14]  Lei Wu,et al.  Compact projection: Simple and efficient near neighbor search with practical memory requirements , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[15]  Bernt Schiele,et al.  Generative Adversarial Text to Image Synthesis , 2016, ICML.

[16]  David S. Ebert,et al.  Structuring Feature Space: A Non-Parametric Method for Volumetric Transfer Function Generation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[17]  Kwan-Liu Ma,et al.  Visibility Histograms and Visibility-Driven Transfer Functions , 2011, IEEE Transactions on Visualization and Computer Graphics.

[18]  Ross T. Whitaker,et al.  Curvature-based transfer functions for direct volume rendering: methods and applications , 2003, IEEE Visualization, 2003. VIS 2003..

[19]  Anders Ynnerman,et al.  Local Histograms for Design of Transfer Functions in Direct Volume Rendering , 2006, IEEE Transactions on Visualization and Computer Graphics.

[20]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Renato Pajarola,et al.  State‐of‐the‐Art in Compressed GPU‐Based Direct Volume Rendering , 2014, Comput. Graph. Forum.

[22]  Fisher Yu,et al.  Scribbler: Controlling Deep Image Synthesis with Sketch and Color , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Thomas Brox,et al.  Learning to generate chairs with convolutional neural networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Stefan Bruckner,et al.  Volume visualization based on statistical transfer-function spaces , 2010, 2010 IEEE Pacific Visualization Symposium (PacificVis).

[25]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Martin Falk,et al.  Intuitive Exploration of Volumetric Data Using Dynamic Galleries , 2016, IEEE Transactions on Visualization and Computer Graphics.

[27]  Xiaoru Yuan,et al.  WYSIWYG (What You See is What You Get) Volume Visualization , 2011, IEEE Transactions on Visualization and Computer Graphics.

[28]  Michael Werman,et al.  Fast and robust Earth Mover's Distances , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[29]  Kwan-Liu Ma,et al.  Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[30]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kwan-Liu Ma,et al.  A novel interface for higher-dimensional classification of volume data , 2003, IEEE Visualization, 2003. VIS 2003..

[32]  Kwan-Liu Ma,et al.  Size-based Transfer Functions: A New Volume Exploration Technique , 2008, IEEE Transactions on Visualization and Computer Graphics.

[33]  Kwan-Liu Ma,et al.  Intelligent Feature Extraction and Tracking for Visualizing Large-Scale 4D Flow Simulations , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[34]  William E. Lorensen,et al.  The Transfer Function Bake-Off , 2001, IEEE Computer Graphics and Applications.

[35]  David Poeppel,et al.  Analysis by Synthesis: A (Re-)Emerging Program of Research for Language and Vision , 2010, Biolinguistics.

[36]  Aaron Knoll,et al.  OSPRay - A CPU Ray Tracing Framework for Scientific Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[37]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[38]  Ulrik Brandes,et al.  Generative Data Models for Validation and Evaluation of Visualization Techniques , 2016, BELIV '16.

[39]  Joe Michael Kniss,et al.  Multidimensional Transfer Functions for Interactive Volume Rendering , 2002, IEEE Trans. Vis. Comput. Graph..

[40]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[41]  Torsten Möller,et al.  Integrating Isosurface Statistics and Histograms , 2013, IEEE Transactions on Visualization and Computer Graphics.

[42]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[43]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[44]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[45]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[46]  Anders Ynnerman,et al.  Correlated Photon Mapping for Interactive Global Illumination of Time-Varying Volumetric Data , 2017, IEEE Transactions on Visualization and Computer Graphics.

[47]  Hans-Christian Hege,et al.  Positional Uncertainty of Isocontours: Condition Analysis and Probabilistic Measures , 2011, IEEE Transactions on Visualization and Computer Graphics.

[48]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[49]  Paul A. Beardsley,et al.  Design galleries: a general approach to setting parameters for computer graphics and animation , 1997, SIGGRAPH.

[50]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.

[51]  Ivan Viola,et al.  Automatic Transfer Functions Based on Informational Divergence , 2011, IEEE Transactions on Visualization and Computer Graphics.

[52]  Nelson L. Max,et al.  Optical Models for Direct Volume Rendering , 1995, IEEE Trans. Vis. Comput. Graph..

[53]  Joe Michael Kniss,et al.  Statistically quantitative volume visualization , 2005, VIS 05. IEEE Visualization, 2005..

[54]  Yingcai Wu,et al.  Interactive Transfer Function Design Based on Editing Direct Volume Rendered Images , 2007, IEEE Transactions on Visualization and Computer Graphics.

[55]  Hanspeter Pfister,et al.  Generation of transfer functions with stochastic search techniques , 1996, Proceedings of Seventh Annual IEEE Visualization '96.

[56]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[57]  Wei Li,et al.  Transfer Function Map , 2014, 2014 IEEE Pacific Visualization Symposium.

[58]  Jeffrey K. Hollingsworth.  Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , 2017, SC.

[59]  Kwan-Liu Ma,et al.  The Occlusion Spectrum for Volume Classification and Visualization , 2009, IEEE Transactions on Visualization and Computer Graphics.

[60]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[61]  Alireza Entezari,et al.  Volumetric Data Reduction in a Compressed Sensing Framework , 2014, Comput. Graph. Forum.