Neural network-based, structure-preserving entropy closures for the Boltzmann moment system

This work presents neural network based minimal entropy closures for the moment system of the Boltzmann equation, that preserve the inherent structure of the system of partial differential equations, such as entropy dissipation and hyperbolicity. The described method embeds convexity of the moment to entropy map in the neural network approximation to preserve the structure of the minimal entropy closure. Two techniques are used to implement the methods. The first approach approximates the map between moments and the minimal entropy of the moment system and is convex by design. The second approach approximates the map between moments and Lagrange multipliers of the dual of the minimal entropy optimization problem, which present the gradients of the entropy with respect to the moments, and is enforced to be monotonic by introduction of a penalty function. We derive an error bound for the generalization gap of convex neural networks which are trained in Sobolev norm and use the results to construct data sampling methods for neural network training. Numerical experiments are conducted, which show that neural network-based entropy closures provide a significant speedup for kinetic solvers while maintaining a sufficient level of accuracy. The code for the described implementations can be found in the Github repositories [1, 2].

[1]  Juntao Huang,et al.  Machine Learning Moment Closure Models for the Radiative Transfer Equation III: Enforcing Hyperbolicity and Physical Characteristic Speeds , 2021, Journal of Scientific Computing.

[2]  C. D. Levermore,et al.  Moment closure hierarchies for kinetic theories , 1996 .

[3]  Kun Xu,et al.  A well-balanced unified gas-kinetic scheme for multiscale flow transport under gravitational field , 2017, J. Comput. Phys..

[4]  T. Konstantin Rusch,et al.  Enhancing accuracy of deep learning algorithms by training with low-discrepancy sequences , 2020, SIAM J. Numer. Anal..

[5]  Edgar Olbrant,et al.  A realizability-preserving discontinuous Galerkin method for the M1 model of radiative transfer , 2012, J. Comput. Phys..

[6]  H. Robbins.  A Stochastic Approximation Method , 1951 .

[7]  Cory D. Hauck,et al.  Data-driven, structure-preserving approximations to entropy-based moment closures for kinetic equations , 2021, Communications in Mathematical Sciences.

[8]  Tianbai Xiao,et al.  Using neural networks to accelerate the solution of the Boltzmann equation , 2020, J. Comput. Phys..

[9]  Razvan Pascanu,et al.  Sobolev Training for Neural Networks , 2017, NIPS.

[10]  T. Luo,et al.  Physics-informed neural networks for solving multiscale mode-resolved phonon Boltzmann transport equation , 2021, Materials Today Physics.

[11]  Kun Xu,et al.  A velocity-space adaptive unified gas kinetic scheme for continuum and rarefied flows , 2018, J. Comput. Phys..

[12]  C. Cercignani.  The Boltzmann equation and its applications , 1988 .

[13]  B. Dubroca,et al.  Etude théorique et numérique d'une hiérarchie de modèles aux moments pour le transfert radiatif , 1999 .

[14]  Chenhao Ma,et al.  Machine learning surrogate models for Landau fluid closure , 2019 .

[15]  Michael Junk,et al.  Domain of Definition of Levermore's Five-Moment System , 1998 .

[16]  J. Lasserre.  Moments, Positive Polynomials And Their Applications , 2009 .

[17]  Klaus-Robert Müller,et al.  Efficient BackProp , 2012, Neural Networks: Tricks of the Trade.

[18]  E. Lewis,et al.  Computational Methods of Neutron Transport , 1993 .

[19]  Ben Adcock,et al.  The gap between theory and practice in function approximation with deep neural networks , 2021, SIAM J. Math. Data Sci..

[20]  Tianbai Xiao.  A flux reconstruction kinetic scheme for the Boltzmann equation , 2021, J. Comput. Phys..

[21]  Michael Junk,et al.  MAXIMUM ENTROPY FOR REDUCED MOMENT PROBLEMS , 2000 .

[22]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[23]  Cory D. Hauck,et al.  High-Order Entropy-Based Closures for Linear Transport in Slab Geometry II: A Computational Study of the Optimization Problem , 2012, SIAM J. Sci. Comput..

[24]  Martin Frank,et al.  Ray effect mitigation for the discrete ordinates method through quadrature rotation , 2018, J. Comput. Phys..

[25]  L. Berkovitz.  Convexity and Optimization in Rn , 2001 .

[26]  Yuanyuan Shi,et al.  Optimal Control Via Neural Networks: A Convex Approach , 2018, ICLR.

[27]  Thierry Goudon,et al.  Analysis of the M1 model: Well-posedness and diffusion asymptotics , 2013 .

[28]  V. Pavan.  General Entropic Approximations for Canonical Systems Described by Kinetic Equations , 2011 .

[29]  Lei Wu,et al.  Machine learning based non-Newtonian fluid model with molecular fidelity , 2020, Physical review. E.

[30]  Lei Wu,et al.  Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't , 2020, CSIAM Transactions on Applied Mathematics.

[31]  C. Schmeiser,et al.  Semiconductor equations , 1990 .

[32]  Moustafa T. Chahine,et al.  Foundations of Radiation Hydrodynamics (Dimitri Mihalas and Barbara Weibel Mihalas) , 1987 .

[33]  Chao Ma,et al.  Uniformly accurate machine learning-based hydrodynamic models for kinetic equations , 2019, Proceedings of the National Academy of Sciences.

[34]  Diego G. Loyola,et al.  Smart sampling and incremental function learning for very large high dimensional data , 2016, Neural Networks.

[35]  Juntao Huang,et al.  Machine learning moment closure models for the radiative transfer equation I: directly learning a gradient based closure , 2021, J. Comput. Phys..

[36]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  R. Molinaro,et al.  Physics Informed Neural Networks for Simulating Radiative Transfer , 2020, Journal of Quantitative Spectroscopy and Radiative Transfer.

[38]  R. Maulik,et al.  Neural network representability of fully ionized plasma fluid model closures , 2020, 2002.04106.

[39]  Emmanuel Franck,et al.  A neural network closure for the Euler-Poisson system based on kinetic simulations , 2020, ArXiv.

[40]  D. S. Kershaw,et al.  Flux limiting nature`s own way -- A new method for numerical solution of the transport equation , 1976 .

[41]  C. David Levermore,et al.  Entropy-based moment closures for kinetic equations , 1997 .

[42]  Martin Frank,et al.  A Regularized Entropy-Based Moment Method for Kinetic Equations , 2018, SIAM J. Appl. Math..

[43]  M. Kreĭn,et al.  The Markov Moment Problem and Extremal Problems , 1977 .

[44]  Cory D. Hauck,et al.  Optimization and large scale computation of an entropy-based moment closure , 2015, J. Comput. Phys..

[45]  Martin Frank,et al.  Moment realizability and Kershaw closures in radiative transfer , 2013 .

[46]  Manuel Torrilhon,et al.  Gaussian Process Regression for Maximum Entropy Distribution , 2020, J. Comput. Phys..

[47]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[48]  Lei Xu,et al.  Input Convex Neural Networks : Supplementary Material , 2017 .

[49]  George Em Karniadakis,et al.  Physics-informed neural networks for solving forward and inverse flow problems via the Boltzmann-BGK formulation , 2020, J. Comput. Phys..

[50]  Michael Junk,et al.  Maximum entropy moment systems and Galilean invariance , 2002 .

[51]  Cory D. Hauck,et al.  A Comparison of Moment Closures for Linear Kinetic Transport Equations: The Line Source Benchmark , 2013 .

[52]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[53]  Thomas A. Brunner,et al.  Forms of Approximate Radiation Transport , 2002 .