Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems

Abstract Simulating complex physical systems often involves solving partial differential equations (PDEs) with some closures due to the presence of multi-scale physics that cannot be fully resolved. Although the advancement of high performance computing has made resolving small-scale physics possible, such simulations are still very expensive. Therefore, reliable and accurate closure models for the unresolved physics remains an important requirement for many computational physics problems, e.g., turbulence simulation. Recently, several researchers have adopted generative adversarial networks (GANs), a novel paradigm of training machine learning models, to generate solutions of PDEs-governed complex systems without having to numerically solve these PDEs. However, GANs are known to be difficult in training and likely to converge to local minima, where the generated samples do not capture the true statistics of the training data. In this work, we present a statistical constrained generative adversarial network by enforcing constraints of covariance from the training data, which results in an improved machine-learning-based emulator to capture the statistics of the training data generated by solving fully resolved PDEs. We show that such a statistical regularization leads to better performance compared to standard GANs, measured by (1) the constrained model's ability to more faithfully emulate certain physical properties of the system and (2) the significantly reduced (by up to 80%) training time to reach the solution. We exemplify this approach on the Rayleigh-Benard convection, a turbulent flow system that is an idealized model of the Earth's atmosphere. With the growth of high-fidelity simulation databases of physical systems, this work suggests great potential for being an alternative to the explicit modeling of closures or parameterizations for unresolved physics, which are known to be a major source of uncertainty in simulating multi-scale physical systems, e.g., turbulence or Earth's climate.

[1]  Hans Johnston,et al.  Comparison of turbulent thermal convection between conditions of constant temperature and constant flux. , 2008, Physical review letters.

[2]  Charles R. Doering,et al.  A comparison of turbulent thermal convection between conditions of constant temperature and constant heat flux boundaries , 2009 .

[3]  Pierre Gentine,et al.  Deep learning to represent subgrid processes in climate models , 2018, Proceedings of the National Academy of Sciences.

[4]  Janet D Scheel,et al.  Turbulent superstructures in Rayleigh-Bénard convection , 2018, Nature Communications.

[5]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[6]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[7]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[8]  Jin-Long Wu,et al.  Enforcing Deterministic Constraints on Generative Adversarial Networks for Emulating Physical Systems , 2019 .

[9]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[10]  Heng Xiao,et al.  Physics-informed covariance kernel for model-form uncertainty quantification with application to turbulent flows , 2019, Computers & Fluids.

[11]  Jacob D. Abernethy,et al.  How to Train Your DRAGAN , 2017, ArXiv.

[12]  Dragos B. Chirila.  Towards lattice Boltzmann models for climate sciences : The GeLB programming language with applications , 2018 .

[13]  Michael Chertkov,et al.  From Deep to Physics-Informed Learning of Turbulence: Diagnostics , 2018, ArXiv.

[14]  Alexandre M. Tartakovsky,et al.  Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks , 2018, J. Comput. Phys..

[15]  Richard J. A. M. Stevens,et al.  Comparison between two- and three-dimensional Rayleigh–Bénard convection , 2013, Journal of Fluid Mechanics.

[16]  Lord Rayleigh,et al.  LIX. On convection currents in a horizontal layer of fluid, when the higher temperature is on the under side , 1916 .

[17]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[18]  Anoop Cherian,et al.  Efficient similarity search for covariance matrices via the Jensen-Bregman LogDet Divergence , 2011, 2011 International Conference on Computer Vision.

[19]  George E. Karniadakis,et al.  Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data , 2018, ArXiv.

[20]  Noah D. Brenowitz,et al.  Prognostic Validation of a Neural Network Unified Physics Parameterization , 2018, Geophysical Research Letters.

[21]  Jinlong Wu,et al.  Physics-informed machine learning approach for reconstructing Reynolds stress modeling discrepancies based on DNS data , 2016, 1606.07987.

[22]  Amy McGovern,et al.  Machine Learning Enhancement of Storm-Scale Ensemble Probabilistic Quantitative Precipitation Forecasts , 2014 .

[23]  Sreenivasan,et al.  Probability density of velocity increments in turbulent flows. , 1992, Physical review letters.

[24]  Karthik Kashinath,et al.  ClimateNet: A Machine Learning dataset for Climate Science Research , 2017 .

[25]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[26]  H. Ozcelik,et al.  Correlation matrix distance, a meaningful measure for evaluation of non-stationary MIMO channels , 2005, 2005 IEEE 61st Vehicular Technology Conference.

[27]  Richard J. A. M. Stevens,et al.  Boundary layer structure in turbulent thermal convection and its consequences for the required numerical resolution , 2010, 1109.6870.

[28]  Quan Zhou,et al.  Statistics of velocity and temperature fluctuations in two-dimensional Rayleigh-Bénard convection. , 2017, Physical review. E.

[29]  E Weinan,et al.  Model Reduction with Memory and the Machine Learning of Dynamical Systems , 2018, Communications in Computational Physics.

[30]  Bin Dong,et al.  PDE-Net: Learning PDEs from Data , 2017, ICML.

[31]  Nils Thürey,et al.  tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow , 2018, ACM Trans. Graph..

[32]  Li Li,et al.  Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds , 2018, ArXiv.

[33]  Dumitru Erhan,et al.  Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Hans Hagen,et al.  Visualization and Processing of Tensor Fields , 2014 .

[35]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[36]  Prabhat,et al.  Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets , 2016, ArXiv.

[37]  G. L. Vasconcelos,et al.  Emergence of skewed non-Gaussian distributions of velocity increments in isotropic turbulence , 2018, Physical Review Fluids.

[38]  Peter Bauer,et al.  Challenges and design choices for global weather and climate models based on machine learning , 2018, Geoscientific Model Development.

[39]  Oliver Hennigh,et al.  Lat-Net: Compressing Lattice Boltzmann Flow Simulations using Deep Neural Networks , 2017, 1705.09036.

[40]  Trevor Darrell,et al.  Discriminator Rejection Sampling , 2018, ICLR.

[41]  Mahendra K. Verma,et al.  Dynamics of reorientations and reversals of large-scale flow in Rayleigh–Bénard convection , 2010, Journal of Fluid Mechanics.

[42]  Andrew Stuart,et al.  Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High‐Resolution Simulations , 2017, 1709.00037.

[43]  Liu Yang,et al.  Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations , 2018, SIAM J. Sci. Comput..

[44]  Prabhat,et al.  ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events , 2016, NIPS.

[45]  Xin Yao,et al.  Evolutionary Generative Adversarial Networks , 2018, IEEE Transactions on Evolutionary Computation.

[46]  Pierre Lallemand,et al.  Lattice Boltzmann simulations of thermal convective flows in two dimensions , 2013, Comput. Math. Appl..

[47]  Bin Dong,et al.  Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations , 2017, ICML.

[48]  Sebastian Nowozin,et al.  The Numerics of GANs , 2017, NIPS.

[49]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[50]  Jinlong Wu,et al.  Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework , 2018, Physical Review Fluids.

[51]  Jacob Abernethy,et al.  On Convergence and Stability of GANs , 2018 .

[52]  Steven L. Brunton,et al.  Deep learning for universal linear embeddings of nonlinear dynamics , 2017, Nature Communications.

[53]  Nagiza F. Samatova,et al.  Theory-Guided Data Science: A New Paradigm for Scientific Discovery from Data , 2016, IEEE Transactions on Knowledge and Data Engineering.

[54]  Deborah Bard,et al.  CosmoGAN: creating high-fidelity weak lensing convergence maps using Generative Adversarial Networks , 2017, Computational Astrophysics and Cosmology.

[55]  J. Templeton,et al.  Reynolds averaged turbulence modelling using deep neural networks with embedded invariance , 2016, Journal of Fluid Mechanics.

[56]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[57]  Léon Bottou,et al.  Wasserstein GAN , 2017, ArXiv.