Depth separation for reduced deep networks in nonlinear model reduction: Distilling shock waves in nonlinear hyperbolic problems

Classical reduced models are low-rank approximations using a fixed basis designed to achieve dimensionality reduction of large-scale systems. In this work, we introduce reduced deep networks, a generalization of classical reduced models formulated as deep neural networks. We prove depth separation results showing that reduced deep networks approximate solutions of parametrized hyperbolic partial differential equations with approximation error $\epsilon$ with $\mathcal{O}(|\log(\epsilon)|)$ degrees of freedom, even in the nonlinear setting where solutions exhibit shock waves. We also show that classical reduced models achieve exponentially worse approximation rates by establishing lower bounds on the relevant Kolmogorov $N$-widths.

[1]  Steven L. Brunton,et al.  Dimensionality reduction and reduced-order modeling for traveling wave physics , 2019, 1911.00565.

[2]  Christoph Schwab,et al.  Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ , 2018, Analysis and Applications.

[3]  Kevin Carlberg,et al.  Adaptive h‐refinement for reduced‐order models , 2014, ArXiv.

[4]  Tao Zhang,et al.  Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and Challenges , 2018, IEEE Signal Processing Magazine.

[5]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[6]  Benjamin Peherstorfer.  Model reduction for transport-dominated problems via online adaptive bases and adaptive sampling , 2020, SIAM J. Sci. Comput..

[7]  Mario Ohlberger,et al.  Reduced Basis Methods: Success, Limitations and Future Challenges , 2015, 1511.02021.

[8]  Donsub Rim,et al.  Transport Reversal for Model Reduction of Hyperbolic Partial Differential Equations , 2017, SIAM/ASA J. Uncertain. Quantification.

[9]  A. Quarteroni,et al.  Reduced Basis Techniques For Nonlinear Conservation Laws , 2015 .

[10]  Gitta Kutyniok,et al.  Numerical Solution of the Parametric Diffusion Equation by Deep Neural Networks , 2020, Journal of Scientific Computing.

[11]  Yixin Chen,et al.  Compressing Neural Networks with the Hashing Trick , 2015, ICML.

[12]  Karen Willcox,et al.  A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems , 2015, SIAM Rev..

[13]  Jean-Frédéric Gerbeau,et al.  Approximated Lax pairs for the reduced order integration of nonlinear evolution equations , 2014, J. Comput. Phys..

[14]  Angelo Iollo,et al.  Advection modes by optimal mass transfer. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[15]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[16]  R. LeVeque.  Finite Difference Methods for Ordinary and Partial Differential Equations: Steady-State and Time-Dependent Problems (Classics in Applied Mathematics Classics in Applied Mathemat) , 2007 .

[17]  Benjamin Stamm,et al.  Model Order Reduction for Problems with Large Convection Effects , 2018, Computational Methods in Applied Sciences.

[18]  Maciej Balajewicz,et al.  Lagrangian basis method for dimensionality reduction of convection dominated nonlinear flows , 2017, ArXiv.

[19]  J. Marsden,et al.  Reconstruction equations and the Karhunen—Loéve expansion for systems with symmetry , 2000 .

[20]  D. Donoho.  Sparse Components of Images and Optimal Atomic Decompositions , 2001 .

[21]  Alexander Novikov,et al.  Tensorizing Neural Networks , 2015, NIPS.

[22]  O. Mula,et al.  Nonlinear model reduction on metric spaces. Application to one-dimensional conservative PDEs in Wasserstein spaces , 2019, ArXiv.

[23]  J. Hesthaven,et al.  Certified Reduced Basis Methods for Parametrized Partial Differential Equations , 2015 .

[24]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[25]  G. Petrova,et al.  Nonlinear Approximation and (Deep) ReLU\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathrm {ReLU}$$\end{document} , 2019, Constructive Approximation.

[26]  Sophia Blau,et al.  Analysis Of The Finite Element Method , 2016 .

[27]  Yi Zhang,et al.  Stronger generalization bounds for deep nets via a compression approach , 2018, ICML.

[28]  Rich Caruana,et al.  Model compression , 2006, KDD '06.

[29]  Gerrit Welper,et al.  Interpolation of Functions with Parameter Dependent Jumps by Transformed Snapshots , 2017, SIAM J. Sci. Comput..

[30]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[31]  David A. McAllester,et al.  A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks , 2017, ICLR.

[32]  Tommaso Taddei,et al.  A registration method for model order reduction: data compression and geometry reduction , 2019, SIAM J. Sci. Comput..

[33]  Benjamin Peherstorfer,et al.  Online Adaptive Model Reduction for Nonlinear Systems via Low-Rank Updates , 2015, SIAM J. Sci. Comput..

[34]  Paris Perdikaris,et al.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations , 2019, J. Comput. Phys..

[35]  Pierre F. J. Lermusiaux,et al.  Dynamically orthogonal field equations for continuous stochastic dynamical systems , 2009 .

[36]  Benjamin Unger,et al.  Nonlinear Galerkin Model Reduction for Systems with Multiple Transport Velocities , 2019, ArXiv.

[37]  Helmut Bölcskei,et al.  Optimal Approximation with Sparsely Connected Deep Neural Networks , 2017, SIAM J. Math. Data Sci..

[38]  Volker Mehrmann,et al.  The Shifted Proper Orthogonal Decomposition: A Mode Decomposition for Multiple Transport Phenomena , 2015, SIAM J. Sci. Comput..

[39]  Fabian Laakmann,et al.  Efficient approximation of solutions of parametric linear transport equations by ReLU DNNs , 2020, Advances in Computational Mathematics.

[40]  Karsten Urban,et al.  Decay of the Kolmogorov N-width for wave problems , 2019, Appl. Math. Lett..

[41]  A. Pinkus.  n-Widths in Approximation Theory , 1985 .

[42]  Vladimir Temlyakov,et al.  CAMBRIDGE MONOGRAPHS ON APPLIED AND COMPUTATIONAL MATHEMATICS , 2022 .

[43]  Kookjin Lee,et al.  Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders , 2018, J. Comput. Phys..

[44]  G. Welper,et al.  Transformed Snapshot Interpolation with High Resolution Transforms , 2019, SIAM J. Sci. Comput..

[45]  Othmar Koch,et al.  Dynamical Low-Rank Approximation , 2007, SIAM J. Matrix Anal. Appl..

[46]  Benjamin Unger,et al.  Projection-based model reduction with dynamically transformed modes , 2019 .

[47]  Donsub Rim,et al.  Dimensional Splitting of Hyperbolic Partial Differential Equations Using the Radon Transform , 2017, SIAM J. Sci. Comput..

[48]  Fabio Nobile,et al.  Symplectic dynamical low rank approximation of wave equations with random parameters , 2020 .

[49]  Qian Wang,et al.  Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem , 2019, J. Comput. Phys..

[50]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[51]  G. M.,et al.  Partial Differential Equations I , 2023, Applied Mathematical Sciences.

[52]  Alfio Quarteroni,et al.  Machine learning for fast and reliable solution of time-dependent differential equations , 2019, J. Comput. Phys..

[53]  Kristian Kirsch,et al.  Theory Of Ordinary Differential Equations , 2016 .

[54]  Mario Ohlberger,et al.  Nonlinear reduced basis approximation of parameterized evolution equations via the method of freezing , 2013 .

[55]  R. LeVeque.  Finite Volume Methods for Hyperbolic Problems: Characteristics and Riemann Problems for Linear Hyperbolic Equations , 2002 .

[56]  Gitta Kutyniok,et al.  A Theoretical Analysis of Deep Neural Networks and Parametric PDEs , 2019, Constructive Approximation.

[57]  Matus Telgarsky,et al.  Benefits of Depth in Neural Networks , 2016, COLT.

[58]  Amir Shahirpour,et al.  A characteristic dynamic mode decomposition , 2016, Theoretical and Computational Fluid Dynamics.

[59]  Gianluigi Rozza,et al.  Overcoming slowly decaying Kolmogorov n-width by transport maps: application to model order reduction of fluid dynamics and fluid-structure interaction problems , 2019, ArXiv.

[60]  Kyle T. Mandli,et al.  Displacement Interpolation Using Monotone Rearrangement , 2017, SIAM/ASA J. Uncertain. Quantification.

[61]  P. Lax.  Hyperbolic Systems of Conservation Laws and the Mathematical Theory of Shock Waves , 1987 .

[62]  Benjamin Peherstorfer,et al.  Manifold Approximations via Transported Subspaces: Model reduction for transport-dominated problems , 2019, ArXiv.

[63]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[64]  Rémi Abgrall,et al.  Robust model reduction by $$L^{1}$$L1-norm minimization and approximation via dictionaries: application to nonlinear hyperbolic problems , 2016, Adv. Model. Simul. Eng. Sci..

[65]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[66]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.