Chaos and Weather Forecasting: the Role of the unstable Subspace in Predictability and State Estimation Problems

In the first part of this paper, we review some important results on atmospheric predictability, from the pioneering work of Lorenz to recent results with operational forecasting models. Particular relevance is given to the connection between atmospheric predictability and the theory of Lyapunov exponents and vectors. In the second part, we briefly review the foundations of data assimilation methods and then we discuss recent results regarding the application of the tools typical of chaotic systems theory described in the first part to well established data assimilation algorithms, the Extended Kalman Filter (EKF) and Four Dimensional Variational Assimilation (4DVar). In particular, the Assimilation in the Unstable Space (AUS), specifically developed for application to chaotic systems, is described in detail.

[1]  C. Nicolis,et al.  Dynamics of Model Error: Some Generic Features , 2003 .

[2]  Luigi Palatella,et al.  Nonlinear Processes in Geophysics On the Kalman Filter error covariance collapse into the unstable subspace , 2011 .

[3]  Arthur Y. Hou,et al.  Empirical Correction of a Dynamical Model. Part I: Fundamental Issues , 1999 .

[4]  Craig H. Bishop,et al.  Ensemble Transformation and Adaptive Observations , 1999 .

[5]  Douglas K. Lilly,et al.  Numerical prediction of thunderstorms—has its time come? , 1990 .

[6]  A. Dalcher,et al.  Forecasting forecast skill , 1987 .

[7]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[8]  Rui A. P. Perdigão,et al.  Dynamics of Prediction Errors under the Combined Effect of Initial Condition and Model Errors , 2009 .

[9]  Joseph Tribbia,et al.  Scale Interactions and Atmospheric Predictability: An Updated Perspective , 2004 .

[10]  R. Vautard,et al.  Observational Error Structures and the Value of Advanced Assimilation Techniques , 2000 .

[11]  Ott,et al.  Observing chaos: Deducing and tracking the state of a chaotic system from limited observation. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[12]  Istvan Szunyogh,et al.  A Local Ensemble Kalman Filter for Atmospheric Data Assimilation , 2002 .

[13]  E. Lorenz.  Atmospheric Predictability as Revealed by Naturally Occurring Analogues , 1969 .

[14]  Malaquias Peña,et al.  Data assimilation and numerical forecasting with imperfect models: The mapping paradigm , 2007 .

[15]  J. Smagorinsky,et al.  GENERAL CIRCULATION EXPERIMENTS WITH THE PRIMITIVE EQUATIONS , 1963 .

[16]  C E Leith,et al.  Objective Methods for Weather Prediction , 1978 .

[17]  R. Daley.  The Effect of Serially Correlated Observation and Model Error on Atmospheric Data Assimilation , 1992 .

[18]  E. Lorenz.  The local structure of a chaotic attractor in four dimensions , 1984 .

[19]  R. Samelson,et al.  An efficient method for recovering Lyapunov vectors from singular vectors , 2007 .

[20]  Christopher M. Danforth,et al.  Accounting for Model Errors in Ensemble Data Assimilation , 2009 .

[21]  Ming Cai,et al.  Bred Vectors of the Zebiak–Cane Model and Their Potential Application to ENSO Predictions , 2003 .

[22]  Alberto Carrassi,et al.  Accounting for Model Error in Variational Data Assimilation: A Deterministic Formulation , 2010 .

[23]  Michael Ghil,et al.  Advanced data assimilation in strongly nonlinear dynamical systems , 1994 .

[24]  Gerald Bierman,et al.  Numerical comparison of discrete Kalman filter algorithms: Orbit Determination case study , 1976, 1976 IEEE Conference on Decision and Control including the 15th Symposium on Adaptive Processes.

[25]  Anna Trevisan,et al.  Transient error growth and local predictability: a study in the Lorenz system , 1995 .

[26]  Thomas M. Hamill,et al.  What Constrains Spread Growth in Forecasts Initialized from Ensemble Kalman Filters , 2011 .

[27]  Anna Trevisan,et al.  Assimilation of Standard and Targeted Observations within the Unstable Subspace of the Observation–Analysis–Forecast Cycle System , 2004 .

[28]  R. Todling,et al.  Suboptimal Schemes for Atmospheric Data Assimilation Based on the Kalman Filter , 1994 .

[29]  P. Courtier,et al.  Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. I: Theory , 2007 .

[30]  E. Kalnay,et al.  Ensemble Forecasting at NCEP and the Breeding Method , 1997 .

[31]  S. Vannitsem,et al.  Model error and sequential data assimilation: A deterministic formulation , 2008, 0802.4217.

[32]  Nils Gustafsson,et al.  Discussion on ‘4D-Var or EnKF?’ , 2007 .

[33]  G. Nicolis,et al.  Finite time behavior of small errors in deterministic chaos and Lyapunov exponents , 1993 .

[34]  Geir Evensen,et al.  The Ensemble Kalman Filter: theoretical formulation and practical implementation , 2003 .

[35]  D. Ruelle,et al.  Ergodic theory of chaos and strange attractors , 1985 .

[36]  T. Palmer,et al.  Stochastic representation of model uncertainties in the ECMWF ensemble prediction system , 2007 .

[37]  Shigeo Yoden,et al.  A Relationship between Local Error Growth and Quasi-stationary States: Case Study in the Lorenz System , 1991 .

[38]  B. Friedland.  Treatment of bias in recursive filtering , 1969 .

[39]  Y. Sasaki.  SOME BASIC FORMALISMS IN NUMERICAL VARIATIONAL ANALYSIS , 1970 .

[40]  M. Ghil,et al.  Data assimilation in meteorology and oceanography , 1991 .

[41]  Eugenia Kalnay,et al.  Ensemble Forecasting at NMC: The Generation of Perturbations , 1993 .

[42]  Anna Trevisan,et al.  Periodic Orbits, Lyapunov Vectors, and Singular Vectors in the Lorenz System , 1998 .

[43]  Olivier Talagrand,et al.  Four‐dimensional variational assimilation in the unstable subspace and the optimal subspace dimension , 2010 .

[44]  E. Lorenz,et al.  The essence of chaos , 1993 .

[45]  P. Houtekamer,et al.  An Adaptive Ensemble Kalman Filter , 2000 .

[46]  Anna Trevisan,et al.  Detecting unstable structures and controlling error growth by assimilation of standard and adaptive observations in a primitive equation ocean model , 2006 .

[47]  C. Nicolis,et al.  Lyapunov Vectors and Error Growth Patterns in a T21L3 Quasigeostrophic Model , 1997 .

[48]  Y. Trémolet.  Model‐error estimation in 4D‐Var , 2007 .

[49]  G. Benettin,et al.  Lyapunov Characteristic Exponents for smooth dynamical systems and for hamiltonian systems; A method for computing all of them. Part 2: Numerical application , 1980 .

[50]  Siegfried D. Schubert,et al.  Dynamical Predictability in a Simple General Circulation Model: Average Error Growth. , 1989 .

[51]  Martin Leutbecher,et al.  A Spectral Stochastic Kinetic Energy Backscatter Scheme and Its Impact on Flow-Dependent Predictability in the ECMWF Ensemble Prediction System , 2009 .

[52]  Two-dimensional turbulence and atmospheric predictability , 1971 .

[53]  R. Daley.  Atmospheric Data Analysis , 1991 .

[54]  C. Danforth,et al.  Estimating and Correcting Global Weather Model Error , 2007 .

[55]  Tim N. Palmer,et al.  Ensemble forecasting , 2008, J. Comput. Phys..

[56]  T. Palmer,et al.  Singular Vectors, Metrics, and Adaptive Observations. , 1998 .

[57]  Olivier Talagrand,et al.  Assimilation of Observations, an Introduction (gtSpecial IssueltData Assimilation in Meteology and Oceanography: Theory and Practice) , 1997 .

[58]  E. Lorenz.  Predictability of Weather and Climate: Predictability – a problem partly solved , 2006 .

[59]  C. Nicolis.  Dynamics of model error: The role of unresolved scales revisited , 2004 .

[60]  E. Kalnay,et al.  C ○ 2007 The Authors , 2006 .

[61]  O. Talagrand,et al.  Evaluation of probabilistic prediction systems for a scalar variable , 2005 .

[62]  Olivier Talagrand,et al.  On extending the limits of variational assimilation in nonlinear chaotic systems , 1996 .

[63]  F. L. Dimet,et al.  Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects , 1986 .

[64]  Brown,et al.  Computing the Lyapunov spectrum of a dynamical system from an observed time series. , 1991, Physical review. A, Atomic, molecular, and optical physics.

[65]  Yuejian Zhu,et al.  Initial perturbations based on the ensemble transform (ET) technique in the NCEP global operational forecast system , 2008 .

[66]  C. Nicolis,et al.  Can error source terms in forecasting models be represented as Gaussian Markov noises? , 2005 .

[67]  C. E. Leith,et al.  Predictability of Turbulent Flows , 1972 .

[68]  C. Nicolis.  Probabilistic Aspects of Error Growth In Atmospheric Dynamics , 1992 .

[69]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[70]  V. I. Oseledec.  A multiplicative ergodic theorem: Lyapunov characteristic num-bers for dynamical systems , 1968 .

[71]  Ichiro Fukumori,et al.  A Partitioned Kalman Filter and Smoother , 2002 .

[72]  Takemasa Miyoshi,et al.  Recent Progress of Data Assimilation Methods in Meteorology(125th Anniversary Issue of the Meteorological Society of Japan) , 2007 .

[73]  Craig H. Bishop,et al.  The THORPEX Interactive Grand Global Ensemble , 2010 .

[74]  Andrew F. Loughe,et al.  The Relationship between Ensemble Spread and Ensemble Mean Skill , 1998 .

[75]  John Harlim,et al.  Convex error growth patterns in a global weather model. , 2005, Physical review letters.

[76]  Istvan Szunyogh,et al.  A local ensemble Kalman filter for atmospheric data assimilation , 2004 .

[77]  Philippe Courtier,et al.  Unified Notation for Data Assimilation : Operational, Sequential and Variational , 1997 .

[78]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[79]  C. Nicolis,et al.  Dynamics of Model Error: The Role of the Boundary Conditions , 2007 .

[80]  Ross N. Bannister,et al.  A review of forecast error covariance statistics in atmospheric variational data assimilation. I: Characteristics and measurements of forecast error covariances , 2008 .

[81]  Edward N. Lorenz.  A look at some details of the growth of initial uncertainties , 2005 .

[82]  Michael K. Tippett,et al.  Conditioning of the Stable, Discrete-Time Lyapunov Operator , 2000, SIAM J. Matrix Anal. Appl..

[83]  Jacques Verron,et al.  A singular evolutive extended Kalman filter for data assimilation in oceanography , 1998 .

[84]  A. Piacentini,et al.  Variational data analysis with control of the forecast bias , 2004 .

[85]  O. Talagrand,et al.  Short-range evolution of small perturbations in a barotropic model , 1988 .

[86]  Alberto Carrassi,et al.  Controlling instabilities along a 3DVar analysis cycle by assimilating in the unstable subspace: a comparison with the EnKF , 2008, 0804.2136.

[87]  E. Lorenz.  Atmospheric predictability experiments with a large numerical model , 1982 .

[88]  M. Pedder,et al.  Atmospheric data analysis: by Roger Daley, Cambridge University Press. Cambridge atmospheric and space science series, 2. 457 pages. Published 1991. Price: £55,-/U.S. $79.50 ISBN 0 521 382157. , 1992 .

[89]  三好 建正.  『Atmospheric Modeling, Data Assimilation and Predictability』, E Kalnay著, Cambridge University Press, 2002, 34lpp, $ 45.00, ISBN : 0-521-79629-6 , 2003 .

[90]  E. Lorenz.  A study of the predictability of a 28-variable atmospheric model , 1965 .

[91]  G. Evensen.  Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics , 1994 .

[92]  Istvan Szunyogh,et al.  A comparison of Lyapunov and optimal vectors in a low-resolution GCM , 1997 .

[93]  A. Trevisan.  Impact of transient error growth on global average predictability measures , 1993 .

[94]  G. Benettin,et al.  Lyapunov Characteristic Exponents for smooth dynamical systems and for hamiltonian systems; a method for computing all of them. Part 1: Theory , 1980 .

[95]  Andrew C. Lorenc,et al.  The potential of the ensemble Kalman filter for NWP—a comparison with 4D‐Var , 2003 .