An adaptive buddy check for observational quality control

An adaptive buddy-check algorithm is presented that adjusts tolerances for suspect observations, based on the variability of surrounding data. The algorithm derives from a statistical hypothesis test combined with maximum-likelihood covariance estimation. Its stability is shown to depend on the initial identification of outliers by a simple background check. The adaptive feature ensures that the final quality-control decisions are not very sensitive to prescribed statistics of first-guess and observation errors, nor on other approximations introduced into the algorithm. The implementation of the algorithm in a global atmospheric data assimilation is described. Its performance is contrasted with that of a non-adaptive buddy check, for the surface analysis of an extreme storm that took place over Europe on 27 December 1999. The adaptive algorithm allowed the inclusion of many important observations that differed greatly from the first guess and that would have been excluded on the basis of prescribed statistics. The analysis of the storm development was much improved as a result of these additional observations.

[1]  H. A. Bedient,et al.  AN EXPERIMENT IN AUTOMATIC DATA PROCESSING , 1957 .

[2]  N. B. Ingleby,et al.  Bayesian quality control using multivariate normal distributions , 1993 .

[3]  D. Dee.  On-line Estimation of Error Covariance Parameters for Atmospheric Data Assimilation , 1995 .

[4]  P. Bergthórsson,et al.  Numerical Weather Map Analysis , 1955 .

[5]  Heikki Järvinen,et al.  Variational quality control , 1999 .

[6]  A. Lorenc.  A Global Three-Dimensional Multivariate Statistical Interpolation Scheme , 1981 .

[7]  William G. Collins,et al.  Comprehensive Hydrostatic Quality Control at the National Meteorological Center , 1990 .

[8]  A. Jazwinski.  Stochastic Processes and Filtering Theory , 1970 .

[9]  L. Takacs,et al.  Dynamical Aspects of Climate Simulations Using the GEOS General Circulation Model , 2022 .

[10]  Dick Dee,et al.  Maximum-Likelihood Estimation of Forecast and Observation Error Covariance Parameters. Part II: Applications , 1999 .

[11]  Dick Dee,et al.  Ooce Note Series on Global Modeling and Data Assimilation Maximum-likelihood Estimation of Forecast and Observation Error Covariance Parameters , 2022 .

[12]  S. Cohn,et al.  Assessing the Effects of Data Selection with the DAO Physical-Space Statistical Analysis System* , 1998 .

[13]  Lippincott Williams Wilkins,et al.  Part I , 1997, Neurology.

[14]  H. Storch,et al.  Statistical Analysis in Climate Research , 2000 .

[15]  Richard Tinker,et al.  Climate Assessment for 1999 , 2000 .

[16]  William G. Collins,et al.  Complex Quality Control of Significant Level Rawinsonde Temperatures , 1998 .

[17]  N. B. Ingleby,et al.  Treatment of Gross Errors Using Maximum Probability Theory , 1992 .

[18]  Lawrence L. Takacs,et al.  Data Assimilation Using Incremental Analysis Updates , 1996 .

[19]  L. Gandin,et al.  Complex Quality Control of Meteorological Observations , 1988 .

[20]  Staff Members,et al.  One Year of Operational Numerical Weather Prediction , 1957 .

[21]  A. Morelli.  Inverse Problem Theory , 2010 .

[22]  E. Lehmann.  Testing Statistical Hypotheses , 1960 .

[23]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[24]  Dick Dee,et al.  Maximum-Likelihood Estimation of Forecast and Observation Error Covariance Parameters. Part I: Methodology , 1999 .

[25]  J. Szpunar,et al.  Part II applications , 2003 .

[26]  R. Daley.  Atmospheric Data Analysis , 1991 .

[27]  Andrew C. Lorenc,et al.  Objective quality control of observations using Bayesian methods. Theory, and a practical implementation , 1988 .