Adaptive Tuning of Numerical Weather Prediction Models: Randomized GCV in Three- and Four-Dimensional Data Assimilation

Abstract In variational data assimilation, optimal ingestion of the observational data, and optimal use of prior physical and statistical information involve the choice of numerous weighting, smoothing, and tuning parameters that control the filtering and merging of diverse sources of information. Generally these weights must be obtained from a partial and imperfect understanding of various sources of errors and are frequently chosen by a combination of historical information, physical reasoning, and trial and error. Generalized cross validation (GCV) has long been one of the methods of choice for choosing certain tuning, smoothing, regularization parameters in ill-posed inverse problems, smoothing, and filtering problems. In theory, it is well suited for the adaptive choice of certain parameters that occur in variational objective analysis and for data assimilation problems that are mathematically equivalent to variational problems. The main drawback of the use of GCV in data assimilation problems was th...

[1]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[2]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[3]  G. Wahba,et al.  Some New Mathematical Methods for Variational Objective Analysis Using Splines and Cross Validation , 1980 .

[4]  G. Wahba.  Spline Interpolation and Smoothing on the Sphere , 1981 .

[5]  G. Wahba.  Variational methods in simultaneous optimum interpolation and initialization , 1982 .

[6]  R. Hoffman.  SASS Wind Ambiguity Removal by Direct Minimization , 1982 .

[7]  G. Wahba.  Erratum: Spline Interpolation and Smoothing on the Sphere , 1982 .

[8]  G. Wahba.  Bayesian "Confidence Intervals" for the Cross-validated Smoothing Spline , 1983 .

[9]  M. Hutchinson,et al.  Monthly estimates of wind speed and wind run for Australia , 1984 .

[10]  P. Speckman.  Spline Smoothing and Optimal Rates of Convergence in Nonparametric Regression Models , 1985 .

[11]  Michael Ghil,et al.  An efficient algorithm for estimating noise covariances in distributed systems , 1985 .

[12]  G. Wahba.  A Comparison of GCV and GML for Choosing the Smoothing Parameter in the Generalized Spline Smoothing Problem , 1985 .

[13]  J. M. Lewis,et al.  The use of adjoint equations to solve a variational adjustment problem with advective constraints , 1985 .

[14]  Ker-Chau Li,et al.  Asymptotic optimality of CL and generalized cross-validation in ridge regression with application to spline smoothing , 1986 .

[15]  A. Hollingsworth,et al.  The statistical structure of short-range forecast errors as determined from radiosonde data Part II: The covariance of height and wind errors , 1986 .

[16]  Anthony Hollingsworth,et al.  The statistical structure of short-range forecast errors as determined from radiosonde data , 1986 .

[17]  Moustafa T. Chahine,et al.  Advances in Remote Sensing Retrieval Methods , 1986 .

[18]  Grace Wahba,et al.  THREE TOPICS IN ILL-POSED PROBLEMS , 1987 .

[19]  G. Wahba,et al.  Gcvpack – routines for generalized cross validation , 1987 .

[20]  A. Lorenc.  Optimal nonlinear objective analysis , 1988 .

[21]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[22]  M. Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines , 1989 .

[23]  A. Girard.  A fast ‘Monte-Carlo cross-validation’ procedure for large least squares problems with noisy data , 1989 .

[24]  G. Wahba,et al.  When is the optimal regularization parameter insensitive to the choice of the loss function , 1990 .

[25]  H. Fleming.  Equivalence of regularization and truncated iteration in the solution of III-posed image reconstruction problems , 1990 .

[26]  G. Wahba.  Regularization and Cross Validation Methods for Nonlinear, Implicit, Ill-posed Inverse Problems , 1990 .

[27]  Dick P. Dee,et al.  Simplified adaptive Kalman filtering for large-scale geophysical models , 1990 .

[28]  G. Wahba.  Spline models for observational data , 1990 .

[29]  P. Courtier,et al.  Variational assimilation of meteorological observations with the direct and adjoint shallow-water equations , 1990 .

[30]  M. A. Kaashoek,et al.  Realization and modelling in system theory , 1990 .

[31]  H. Mitchell,et al.  Revised Interpolation Statistics for the Canadian Data Assimilation Procedure: Their Derivation and Application , 1990 .

[32]  D. Girard,et al.  Fast Computation of Cross-Validated Robust Splines and Other Non-linear Smoothing Splines , 1991, Curves and Surfaces.

[33]  Chong Gu,et al.  Minimizing GCV/GML Scores with Multiple Smoothing Parameters via the Newton Method , 1991, SIAM J. Sci. Comput..

[34]  R. S. Bell,et al.  The Meteorological Office analysis correction data assimilation scheme , 1991 .

[35]  Finbarr O'Sullivan.  Sensitivity Analysis for Regularized Estimation in Some System Identification Problems , 1991, SIAM J. Sci. Comput..

[36]  Robert N. Miller,et al.  Weighting Initial Conditions in Variational Assimilation Schemes , 1991 .

[37]  D. Girard.  Asymptotic Optimality of the Fast Randomized Versions of GCV and $C_L$ in Ridge Regression and Regularization , 1991 .

[38]  Ionel M. Navon,et al.  Incomplete observations and control of gravity waves in variational data assimilation , 1992 .

[39]  John Derber,et al.  The National Meteorological Center's spectral-statistical interpolation analysis system , 1992 .

[40]  R. Daley.  Estimating Model-Error Covariances for Application to Atmospheric Data Assimilation , 1992 .

[41]  R. Daley.  The Effect of Serially Correlated Observation and Model Error on Atmospheric Data Assimilation , 1992 .

[42]  H. Mitchell,et al.  A continuous three‐dimensional model of short‐range forecast error covariances , 1992 .

[43]  R. Daley.  The Lagged Innovation Covariance: A Performance Diagnostic for Atmospheric Data Assimilation , 1992 .

[44]  A. Lorenc.  Iterative analysis using covariance functions and filters , 1992 .

[45]  R. Daley.  Forecast-error statistics for homogeneous and inhomogeneous observation networks , 1992 .

[46]  S. Cohn.  Dynamics of Short-Term Univariate Forecast Error Covariances , 1993 .

[47]  Ionel M. Navon,et al.  Control Of Gravitational Oscillations in Variational Data Assimilation , 1993 .

[48]  P. Courtier,et al.  Variational Data Assimilation with a Semi-Lagrangian Semi-implicit Global Shallow-Water Equation Model and Its Adjoint , 1993 .

[49]  J. Goerss,et al.  The Multivariate Optimum Interpolation Analysis of Meteorological Data at the Fleet Numerical Oceanography Center , 1993 .

[50]  Feng Gao,et al.  Fitting smoothing splines to data from multiple sources , 1994 .

[51]  D. Dee.  On-line Estimation of Error Covariance Parameters for Atmospheric Data Assimilation , 1995 .

[52]  C.,et al.  Analysis methods for numerical weather prediction , 2022 .