Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

We present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. The effectiveness our approach is demonstrated through a series of canonical studies, including the regression of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty propagation in high-dimensional dynamical systems.

[1]  Markus Schöberl,et al.  Predictive Collective Variable Discovery with Deep Bayesian Models , 2018, The Journal of chemical physics.

[2]  Michael S. Eldred,et al.  Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust , 2014 .

[3]  Phaedon-Stelios Koutsourelakis,et al.  A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime , 2019, J. Comput. Phys..

[4]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[5]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[6]  Wolfgang Hörmann,et al.  Quasi Importance Sampling , 2005 .

[7]  Gregory P. Way,et al.  Extracting a Biologically Relevant Latent Space from Cancer Transcriptomes with Variational Autoencoders , 2017, bioRxiv.

[8]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[9]  Shakir Mohamed,et al.  Distribution Matching in Variational Inference , 2018, ArXiv.

[10]  C. Villani Optimal Transport: Old and New , 2008 .

[11]  Barnabás Póczos,et al.  Enabling Dark Energy Science with Deep Generative Models of Galaxy Images , 2016, AAAI.

[12]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[13]  Navdeep Jaitly,et al.  Adversarial Autoencoders , 2015, ArXiv.

[14]  Dongbin Xiu,et al.  Computational Aspects of Stochastic Collocation with Multifidelity Models , 2014, SIAM/ASA J. Uncertain. Quantification.

[15]  Dilin Wang,et al.  Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm , 2016, NIPS.

[16]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[17]  Tiangang Cui,et al.  Multifidelity importance sampling , 2016 .

[18]  Guoyin Wang,et al.  Learning to Sample with Adversarially Learned Likelihood-Ratio , 2018 .

[19]  Sebastian Nowozin,et al.  Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks , 2017, ICML.

[20]  Nando de Freitas,et al.  Taking the Human Out of the Loop: A Review of Bayesian Optimization , 2016, Proceedings of the IEEE.

[21]  Nir Yosef,et al.  A deep generative model for single-cell RNA sequencing with application to detecting differentially expressed genes , 2017, ArXiv.

[22]  Michalis K. Titsias,et al.  Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions , 2017, 1708.01529.

[23]  Theresa Dawn Robinson,et al.  Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping , 2008 .

[24]  Benjamin Peherstorfer,et al.  Optimal Model Management for Multifidelity Monte Carlo Estimation , 2016, SIAM J. Sci. Comput..

[25]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[26]  Daniele Venturi,et al.  Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets , 2016, SIAM J. Sci. Comput..

[27]  Benjamin Peherstorfer,et al.  Survey of multifidelity methods in uncertainty propagation, inference, and optimization , 2018, SIAM Rev..

[28]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[29]  Stefano Ermon,et al.  InfoVAE: Balancing Learning and Inference in Variational Autoencoders , 2019, AAAI.

[30]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[31]  Qing Li,et al.  A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials , 2010 .

[32]  Ilias Bilionis,et al.  Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification , 2013, J. Comput. Phys..

[33]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[34]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[35]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[36]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[37]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[38]  Chunyuan Li,et al.  Towards Better Representations with Deep/Bayesian Learning , 2018 .

[39]  Lawrence Carin,et al.  Symmetric Variational Autoencoder and Connections to Adversarial Learning , 2017, AISTATS.

[40]  Youssef M. Marzouk,et al.  Bayesian inference with optimal maps , 2011, J. Comput. Phys..

[41]  Roger B. Grosse,et al.  Isolating Sources of Disentanglement in Variational Autoencoders , 2018, NeurIPS.

[42]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[43]  J. Burgers A mathematical model illustrating the theory of turbulence , 1948 .

[44]  A. O'Hagan,et al.  Predicting the output from a complex computer code when fast approximations are available , 2000 .

[45]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[46]  Guangyao Li,et al.  Multi-fidelity optimization for sheet metal forming process , 2011 .

[47]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[48]  Alexander I. J. Forrester,et al.  Multi-fidelity optimization via surrogate modelling , 2007, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[49]  Ryan P. Adams,et al.  Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach. , 2016, Nature materials.

[50]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[51]  Alán Aspuru-Guzik,et al.  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules , 2016, ACS central science.

[52]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[53]  Stéphane Mallat,et al.  Understanding deep convolutional networks , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[54]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[55]  Bernhard Schölkopf,et al.  Wasserstein Auto-Encoders , 2017, ICLR.

[56]  O. Bousquet,et al.  From optimal transport to generative modeling: the VEGAN cookbook , 2017, 1705.07642.

[57]  Justin Domke,et al.  Importance Weighting and Variational Inference , 2018, NeurIPS.

[58]  David Duvenaud,et al.  Joint Importance Sampling for Variational Inference , 2018 .

[59]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[60]  P. A. Newman,et al.  Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models , 2001 .

[61]  Seungho Lee,et al.  DDDAS-based multi-fidelity simulation framework for supply chain systems , 2010 .

[62]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[63]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[64]  Lloyd N. Trefethen,et al.  Fourth-Order Time-Stepping for Stiff PDEs , 2005, SIAM J. Sci. Comput..

[65]  Dongbin Xiu,et al.  A Stochastic Collocation Algorithm with Multifidelity Models , 2014, SIAM J. Sci. Comput..

[66]  Ivor W. Tsang,et al.  Degeneration in VAE: in the Light of Fisher Information Loss , 2018, ArXiv.

[67]  Max Welling,et al.  Improved Variational Inference with Inverse Autoregressive Flow , 2016, NIPS 2016.

[68]  W. Wall,et al.  Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme , 2014, Biomechanics and Modeling in Mechanobiology.

[69]  G. Karniadakis,et al.  Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond , 2016, Journal of The Royal Society Interface.

[70]  Ruslan Salakhutdinov,et al.  Importance Weighted Autoencoders , 2015, ICLR.

[71]  Daniele Venturi,et al.  Multi-fidelity Gaussian process regression for prediction of random fields , 2017, J. Comput. Phys..

[72]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[73]  Marco Cuturi,et al.  GAN and VAE from an Optimal Transport Point of View , 2017, 1706.01807.

[74]  M. Eldred,et al.  Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification , 2009 .

[75]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[76]  Andreas C. Damianou,et al.  Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling , 2017, Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences.