Conditional Gaussian Nonlinear System: a Fast Preconditioner and a Cheap Surrogate Model For Complex Nonlinear Systems

Developing suitable approximate models for analyzing and simulating complex nonlinear systems is practically important. This paper aims at exploring the skill of a rich class of nonlinear stochastic models, known as the conditional Gaussian nonlinear system (CGNS), as both a cheap surrogate model and a fast preconditioner for facilitating many computationally challenging tasks. The CGNS preserves the underlying physics to a large extent and can reproduce intermittency, extreme events and other non-Gaussian features in many complex systems arising from practical applications. Three interrelated topics are studied. First, the closed analytic formulae of solving the conditional statistics provide an efficient and accurate data assimilation scheme. It is shown that the data assimilation skill of a suitable CGNS approximate forecast model outweighs that by applying an ensemble method even to the perfect model with strong nonlinearity, where the latter suffers from filter divergence. Second, the CGNS allows the development of a fast algorithm for simultaneously estimating the parameters and the unobserved variables with uncertainty quantification in the presence of only partial observations. Utilizing an appropriate CGNS as a preconditioner significantly reduces the computational cost in accurately estimating the parameters in the original complex system. Finally, the CGNS advances rapid and statistically accurate algorithms for computing the probability density function and sampling the trajectories of the unobserved state variables. These fast algorithms facilitate the development of an efficient and accurate data-driven method for predicting the linear response of the original system with respect to parameter perturbations based on a suitable CGNS preconditioner.

[1]  Geoffrey E. Hinton,et al.  Parameter estimation for linear dynamical systems , 1996 .

[2]  Andrew J. Majda,et al.  An ensemble Kalman filter for statistical estimation of physics constrained nonlinear regression models , 2014, J. Comput. Phys..

[3]  Sarah A. Sheard,et al.  Principles of complex systems for systems engineering , 2007, Syst. Eng..

[4]  David A. Freedman,et al.  Statistical Models: Theory and Practice: References , 2005 .

[5]  Karthik Kashinath,et al.  Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers , 2021, ArXiv.

[6]  Eric Vanden-Eijnden,et al.  A computational strategy for multiscale systems with applications to Lorenz 96 model , 2004 .

[7]  Traian Iliescu,et al.  Data-Driven Filtered Reduced Order Modeling of Fluid Flows , 2017, SIAM J. Sci. Comput..

[8]  Changhong Mou,et al.  Reduced Order Models for the Quasi-Geostrophic Equations: A Brief Survey , 2020, Fluids.

[9]  Reduced-order models for coupled dynamical systems: Data-driven methods and the Koopman operator. , 2020, Chaos.

[10]  Peter J. Webster,et al.  Climate Science and the Uncertainty Monster , 2011 .

[11]  Valerio Lucarini,et al.  A proof of concept for scale‐adaptive parametrizations: the case of the Lorenz '96 model , 2016, 1612.07223.

[12]  Dmitri Kondrashov,et al.  Data-adaptive harmonic spectra and multilayer Stuart-Landau models. , 2017, Chaos.

[13]  Mohammad Farazmand,et al.  Extreme Events: Mechanisms and Prediction , 2018, Applied Mechanics Reviews.

[14]  A. Dembo,et al.  Parameter estimation of partially observed continuous time stochastic processes via the EM algorithm , 1992 .

[15]  Andrew J. Majda,et al.  Lessons in uncertainty quantification for turbulent dynamical systems , 2012 .

[16]  Gilead Tadmor,et al.  Reduced-Order Modelling for Flow Control , 2013 .

[17]  Michael Ghil,et al.  Multilevel Regression Modeling of Nonlinear Processes: Derivation and Applications to Climatic Variability , 2005 .

[18]  Michael Ghil,et al.  Predicting stochastic systems by noise sampling, and application to the El Niño-Southern Oscillation , 2011, Proceedings of the National Academy of Sciences.

[19]  Zhong Yi Wan,et al.  Reduced-space Gaussian Process Regression for data-driven probabilistic forecast of chaotic dynamical systems , 2016, 1611.01583.

[20]  Sebastian Reich,et al.  An ensemble Kalman-Bucy filter for continuous data assimilation , 2012 .

[21]  Andrew J. Majda,et al.  Predicting the cloud patterns of the Madden‐Julian Oscillation through a low‐order nonlinear stochastic model , 2014 .

[22]  G. C. Wei,et al.  A Monte Carlo Implementation of the EM Algorithm and the Poor Man's Data Augmentation Algorithms , 1990 .

[23]  Andrew J Majda,et al.  Concrete ensemble Kalman filters with rigorous catastrophic filter divergence , 2015, Proceedings of the National Academy of Sciences.

[24]  Yingda Li,et al.  BAMCAFE: A Bayesian machine learning advanced forecast ensemble method for complex turbulent systems with partial observations. , 2021, Chaos.

[25]  Simo Särkkä,et al.  Bayesian Filtering and Smoothing , 2013, Institute of Mathematical Statistics textbooks.

[26]  Alexandre J. Chorin,et al.  Discrete approach to stochastic parametrization and dimension reduction in nonlinear dynamics , 2015, Proceedings of the National Academy of Sciences.

[27]  Andrew J Majda,et al.  An applied mathematics perspective on stochastic modelling for climate , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[28]  B. R. Noack Turbulence, Coherent Structures, Dynamical Systems and Symmetry , 2013 .

[29]  Valerio Lucarini,et al.  Predicting Climate Change Using Response Theory: Global Averages and Spatial Patterns , 2015, 1512.06542.

[30]  J. McWilliams,et al.  Stochastic rectification of fast oscillations on slow manifold closures , 2021, Proceedings of the National Academy of Sciences.

[31]  Andrew J. Majda,et al.  Information theory and stochastics for multiscale nonlinear systems , 2005 .

[33]  Timothy DelSole,et al.  Predictability and Information Theory. Part II: Imperfect Forecasts , 2005 .

[34]  Andrew J. Majda,et al.  A mathematical framework for stochastic climate models , 2001 .

[35]  Paul N. Edwards,et al.  History of climate modeling , 2011 .

[36]  D. Wilcox Multiscale model for turbulent flows , 1986 .

[37]  Andrew J. Majda,et al.  Non-Gaussian Test Models for Prediction and State Estimation with Model Errors , 2013, Chinese Annals of Mathematics, Series B.

[38]  Tim Palmer,et al.  A Nonlinear Dynamical Perspective on Climate Prediction , 1999 .

[39]  G. Roberts,et al.  Data Augmentation for Diffusions , 2013 .

[40]  P. Fearnhead,et al.  Improved particle filter for nonlinear problems , 1999 .

[41]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[42]  Duane E. Waliser,et al.  Intraseasonal Variability in the Atmosphere-Ocean Climate System , 2005 .

[43]  Christopher K. Wikle,et al.  Atmospheric Modeling, Data Assimilation, and Predictability , 2005, Technometrics.

[44]  E. Kalnay,et al.  Ensemble Forecasting at NCEP and the Breeding Method , 1997 .

[45]  A. Shiryayev,et al.  Statistics of Random Processes Ii: Applications , 2000 .

[46]  Gianluigi Rozza,et al.  Data-Driven POD-Galerkin Reduced Order Model for Turbulent Flows , 2019, J. Comput. Phys..

[47]  D. Nychka Data Assimilation” , 2006 .

[48]  Arno Solin,et al.  Expectation maximization based parameter estimation by sigma-point and particle smoothing , 2014, 17th International Conference on Information Fusion (FUSION).

[49]  Andrew J. Majda,et al.  Physics constrained nonlinear regression models for time series , 2012 .

[50]  Zoubin Ghahramani,et al.  Learning Nonlinear Dynamical Systems Using an EM Algorithm , 1998, NIPS.

[51]  Andrew J. Majda,et al.  Information barriers for noisy Lagrangian tracers in filtering random incompressible flows , 2014 .

[52]  Omer San,et al.  Extreme learning machine for reduced order modeling of turbulent geophysical flows. , 2018, Physical review. E.

[53]  Marc Bocquet,et al.  Data Assimilation: Methods, Algorithms, and Applications , 2016 .

[54]  H. Dijkstra Nonlinear Climate Dynamics , 2013 .

[55]  K. Trenberth,et al.  Attribution of climate extreme events , 2015 .

[56]  Richard Kleeman,et al.  Information Theory and Dynamical System Predictability , 2011, Entropy.

[57]  Omer San,et al.  Data-driven recovery of hidden physics in reduced order modeling of fluid flows , 2020, Physics of Fluids.

[58]  Nan Chen,et al.  Conditional Gaussian Systems for Multiscale Nonlinear Stochastic Systems: Prediction, State Estimation and Uncertainty Quantification , 2018, Entropy.

[59]  Andrew J. Majda,et al.  Test models for improving filtering with model errors through stochastic parameter estimation , 2010, J. Comput. Phys..

[60]  Andrew J. Majda,et al.  Improving filtering and prediction of spatially extended turbulent systems with model errors through stochastic parameter estimation , 2010, J. Comput. Phys..

[61]  Andrew J. Majda,et al.  Beating the curse of dimension with accurate statistics for the Fokker–Planck equation in complex turbulent systems , 2017, Proceedings of the National Academy of Sciences.

[62]  Paul Manneville,et al.  Intermittency and the Lorenz model , 1979 .

[63]  Dirk P. Kroese,et al.  Kernel density estimation via diffusion , 2010, 1011.2602.

[64]  Bjørn Eraker MCMC Analysis of Diffusion Models With Application to Finance , 2001 .

[65]  Tim N. Palmer,et al.  Ensemble forecasting , 2008, J. Comput. Phys..

[66]  I. Moroz,et al.  Stochastic parametrizations and model uncertainty in the Lorenz ’96 system , 2013, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[67]  Nan Chen,et al.  Filtering Nonlinear Turbulent Dynamical Systems through Conditional Gaussian Statistics , 2016 .

[68]  G. Vallis Atmospheric and Oceanic Fluid Dynamics: Fundamentals and Large-Scale Circulation , 2017 .

[69]  K. Hasselmann PIPs and POPs: The reduction of complex dynamical systems using principal interaction and oscillation patterns , 1988 .

[70]  Andrew J Majda,et al.  Mathematical test models for superparametrization in anisotropic turbulence , 2009, Proceedings of the National Academy of Sciences.

[71]  I. Mezić,et al.  Spectral analysis of nonlinear flows , 2009, Journal of Fluid Mechanics.

[72]  M. Ghil,et al.  Data assimilation in meteorology and oceanography , 1991 .

[73]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[74]  Dario Ambrosini,et al.  Data-driven model predictive control using random forests for building energy optimization and climate control , 2018, Applied Energy.

[75]  Valerio Lucarini,et al.  Multi-level Dynamical Systems: Connecting the Ruelle Response Theory and the Mori-Zwanzig Approach , 2012, Journal of Statistical Physics.

[76]  A. Chattopadhyay,et al.  Data‐Driven Super‐Parameterization Using Deep Learning: Experimentation With Multiscale Lorenz 96 Systems and Transfer Learning , 2020, Journal of Advances in Modeling Earth Systems.

[77]  Nan Chen,et al.  Model Error, Information Barriers, State Estimation and Prediction in Complex Multiscale Systems , 2018, Entropy.

[78]  I. J. Myung,et al.  Tutorial on maximum likelihood estimation , 2003 .

[79]  Adrian Sandu,et al.  Efficient Construction of Local Parametric Reduced Order Models Using Machine Learning Techniques , 2015, ArXiv.

[80]  Andrew J. Majda,et al.  A mechanism for catastrophic filter divergence in data assimilation for sparse observation networks , 2013 .

[81]  Sten Bay Jørgensen,et al.  Parameter estimation in stochastic grey-box models , 2004, Autom..

[82]  Fei Lu,et al.  Data-driven model reduction, Wiener projections, and the Koopman-Mori-Zwanzig formalism , 2019, J. Comput. Phys..

[83]  S. Swain Handbook of Stochastic Methods for Physics, Chemistry and the Natural Sciences , 1984 .

[84]  K. Kashinath,et al.  Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence , 2020, CI.

[85]  Andrew J. Majda,et al.  Efficient nonlinear optimal smoothing and sampling algorithms for complex turbulent nonlinear dynamical systems with partial observations , 2020, J. Comput. Phys..

[86]  Mickaël D. Chekroun,et al.  Stochastic parameterizing manifolds and non-markovian reduced equations : stochastic manifolds for nonlinear SPDEs II/ Mickaël D. Chekroun, Honghu Liu, Shouhong Wang , 2014 .

[87]  Valerio Lucarini,et al.  A new framework for climate sensitivity and prediction: a modelling perspective , 2014, Climate Dynamics.

[88]  A. Majda Challenges in Climate Science and Contemporary Applied Mathematics , 2012 .

[89]  Andrew J. Majda,et al.  Predicting Monsoon Intraseasonal Precipitation using a Low-Order Nonlinear Stochastic Model , 2018 .

[90]  Timothy DelSole,et al.  Predictability and Information Theory. Part I: Measures of Predictability , 2004 .

[91]  Benjamin Peherstorfer,et al.  Dynamic data-driven reduced-order models , 2015 .

[92]  Andrew J. Majda,et al.  Stochastic superparameterization in quasigeostrophic turbulence , 2013, J. Comput. Phys..

[93]  O. San,et al.  Data-Driven Variational Multiscale Reduced Order Models , 2020, ArXiv.

[94]  Nan Chen,et al.  Rigorous Analysis for Efficient Statistically Accurate Algorithms for Solving Fokker-Planck Equations in Large Dimensions , 2017, SIAM/ASA J. Uncertain. Quantification.

[95]  G. Evensen Data Assimilation: The Ensemble Kalman Filter , 2006 .

[96]  K. Vahala Handbook of stochastic methods for physics, chemistry and the natural sciences , 1986, IEEE Journal of Quantum Electronics.

[97]  Andrew J. Majda,et al.  Stochastic superparameterization in a one-dimensional model for wave turbulence , 2014 .

[98]  Andrew J. Majda,et al.  New perspectives on superparameterization for geophysical turbulence , 2014, J. Comput. Phys..

[99]  A. Majda Introduction to PDEs and Waves in Atmosphere and Ocean , 2003 .

[100]  D. Randall,et al.  A Multiscale Modeling System: Developments, Applications, and Critical Issues , 2009 .

[101]  Matthew C. Coleman,et al.  Bayesian parameter estimation with informative priors for nonlinear systems , 2006 .

[102]  R. Salmon,et al.  Lectures on Geophysical Fluid Dynamics , 1998 .

[103]  Andrew J. Majda,et al.  Filtering Complex Turbulent Systems , 2012 .

[104]  Luc Van Gool,et al.  An adaptive color-based particle filter , 2003, Image Vis. Comput..

[105]  Tim N. Palmer,et al.  A nonlinear dynamical perspective on climate change , 1993 .

[106]  Darren J. Wilkinson,et al.  Bayesian inference for nonlinear multivariate diffusion models observed with error , 2008, Comput. Stat. Data Anal..

[107]  Michael Ghil,et al.  Data-driven non-Markovian closure models , 2014, 1411.4700.

[108]  Nan Chen,et al.  Efficient statistically accurate algorithms for the Fokker-Planck equation in large dimensions , 2017, J. Comput. Phys..

[109]  Frank Kwasniok,et al.  The reduction of complex dynamical systems using principal interaction patterns , 1996 .

[110]  S. Childress,et al.  Topics in geophysical fluid dynamics. Atmospheric dynamics, dynamo theory, and climate dynamics. , 1987 .

[111]  R. E. Kalman,et al.  New Results in Linear Filtering and Prediction Theory , 1961 .

[112]  Andrew J. Majda,et al.  Low-Frequency Climate Response and Fluctuation–Dissipation Theorems: Theory and Practice , 2010 .

[113]  Charbel Farhat,et al.  The GNAT method for nonlinear model reduction: Effective implementation and application to computational fluid dynamics and turbulent flows , 2012, J. Comput. Phys..

[114]  S. Brunton,et al.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems , 2015, Proceedings of the National Academy of Sciences.

[115]  P. Schmid,et al.  Dynamic mode decomposition of numerical and experimental data , 2008, Journal of Fluid Mechanics.

[116]  Nan Chen,et al.  Learning nonlinear turbulent dynamics from partial observations via analytically solvable conditional statistics , 2020, J. Comput. Phys..

[117]  Eric Vanden-Eijnden,et al.  Subgrid-Scale Parameterization with Conditional Markov Chains , 2008 .

[118]  Vanden Eijnden E,et al.  Models for stochastic climate prediction. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[119]  Andrew J. Majda,et al.  Introduction to Turbulent Dynamical Systems in Complex Systems , 2016 .

[120]  Fredrik Gustafsson,et al.  On Resampling Algorithms for Particle Filters , 2006, 2006 IEEE Nonlinear Statistical Signal Processing Workshop.

[122]  P. N. Edwards GLOBAL CLIMATE SCIENCE, UNCERTAINTY AND POLITICS: DATA-LADEN MODELS, MODEL-FILTERED DATA , 1999 .

[123]  Xin Yan,et al.  Linear Regression Analysis: Theory and Computing , 2009 .

[124]  Karthik Duraisamy,et al.  Modal Analysis of Fluid Flows: Applications and Outlook , 2019, AIAA Journal.

[125]  Extreme events in turbulent flow , 2021, Journal of Fluid Mechanics.