Hidden Error Variance Theory. Part I: Exposition and Analytic Model

AbstractA conundrum of predictability research is that while the prediction of flow-dependent error distributions is one of its main foci, chaos fundamentally hides flow-dependent forecast error distributions from empirical observation. Empirical estimation of such error distributions requires a large sample of error realizations given the same flow-dependent conditions. However, chaotic elements of the flow and the observing network make it impossible to collect a large enough conditioned error sample to empirically define such distributions and their variance. Such conditional variances are “hidden.” Here, an exposition of the problem is developed from an ensemble Kalman filter data assimilation system applied to a 10-variable nonlinear chaotic model and 25 000 replicate models. The 25 000 replicates reveal the error variances that would otherwise be hidden. It is found that the inverse-gamma distribution accurately approximates the posterior distribution of conditional error variances given an imperfec...

[1]  Eugenia Kalnay,et al.  Ensemble Forecasting at NMC: The Generation of Perturbations , 1993 .

[2]  Istvan Szunyogh,et al.  A Local Ensemble Kalman Filter for Atmospheric Data Assimilation , 2002 .

[3]  Eugenia Kalnay,et al.  Atmospheric Modeling, Data Assimilation and Predictability , 2002 .

[4]  P. Houtekamer,et al.  A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation , 2001 .

[5]  Istvan Szunyogh,et al.  Can an ensemble transform Kalman filter predict the reduction in forecast-error variance produced by targeted observations? , 2001 .

[6]  Craig H. Bishop,et al.  Evaluation of the Ensemble Transform Analysis Perturbation Scheme at NRL , 2008 .

[7]  P. L. Houtekamer,et al.  A System Simulation Approach to Ensemble Prediction , 1996 .

[8]  Jeffrey L. Anderson.  An Ensemble Adjustment Kalman Filter for Data Assimilation , 2001 .

[9]  E. Kalnay,et al.  Ensemble Forecasting at NCEP and the Breeding Method , 1997 .

[10]  J. Whitaker,et al.  Ensemble Square Root Filters , 2003, Statistical Methods for Climate Scientists.

[11]  J. Pedlosky,et al.  Chaotic and Periodic Behavior of Finite-Amplitude Baroclinic Waves , 1980 .

[12]  C. Bishop,et al.  Improvement of ensemble reliability with a new dressing kernel , 2005 .

[13]  Xuguang Wang,et al.  A Comparison of Breeding and Ensemble Transform Kalman Filter Ensemble Forecast Schemes , 2003 .

[14]  T. Hamill,et al.  A Hybrid Ensemble Kalman Filter-3D Variational Analysis Scheme , 2000 .

[15]  Tim N. Palmer,et al.  Ensemble forecasting , 2008, J. Comput. Phys..

[16]  Istvan Szunyogh,et al.  Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter , 2005, physics/0511236.

[17]  F. Molteni,et al.  The ECMWF Ensemble Prediction System: Methodology and validation , 1996 .

[18]  Peter Jan,et al.  Particle Filtering in Geophysical Systems , 2009 .

[19]  Craig H. Bishop,et al.  Adaptive sampling with the ensemble transform Kalman filter , 2001 .

[20]  Craig H. Bishop,et al.  Hidden Error Variance Theory. Part II: An Instrument That Reveals Hidden Error Variance Distributions from Ensemble Forecasts and Observations , 2013 .

[21]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[22]  Edward N. Lorenz,et al.  Designing Chaotic Models , 2005 .

[23]  J. Whitaker,et al.  Ensemble Data Assimilation without Perturbed Observations , 2002 .

[24]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .