Estimation Methods for Nonhomogeneous Regression Models: Minimum Continuous Ranked Probability Score versus Maximum Likelihood

Nonhomogeneous regression models are widely used to statistically postprocess numerical ensemble weather prediction models. Such regression models are capable of forecasting full probability distributions and correcting for ensemble errors in the mean and variance. To estimate the corresponding regression coefficients, minimization of the continuous ranked probability score (CRPS) has widely been used in meteorological postprocessing studies and has often been found to yield more calibrated forecasts compared to maximum likelihood estimation. From a theoretical perspective, both estimators are consistent and should lead to similar results, provided the correct distribution assumption about empirical data. Differences between the estimated values indicate a wrong specification of the regression model. This study compares the two estimators for probabilistic temperature forecasting with nonhomogeneous regression, where results show discrepancies for the classical Gaussian assumption. The heavy-tailed logistic and Student’s t distributions can improve forecast performance in terms of sharpness and calibration, and lead to only minor differences between the estimators employed. Finally, a simulation study confirms the importance of appropriate distribution assumptions and shows that for a correctly specified model the maximum likelihood estimator is slightly more efficient than the CRPS estimator.

[1]  Student,et al.  THE PROBABLE ERROR OF A MEAN , 1908 .

[2]  E. Lorenz Deterministic nonperiodic flow , 1963 .

[3]  P. J. Huber The behavior of maximum likelihood estimates under nonstandard conditions , 1967 .

[4]  C. Leith Theoretical Skill of Monte Carlo Forecasts , 1974 .

[5]  Halbert White,et al.  Estimation, inference, and specification analysis , 1996 .

[6]  Jeffrey L. Anderson A Method for Producing and Evaluating Probabilistic Forecasts from Ensemble Model Integrations , 1996 .

[7]  J. Aldrich R.A. Fisher and the making of maximum likelihood 1912-1922 , 1997 .

[8]  T. Hamill,et al.  Evaluation of Eta-RSM Ensemble Probabilistic Precipitation Forecasts , 1998 .

[9]  R. Selten Axiomatic Characterization of the Quadratic Scoring Rule , 1998 .

[10]  H. Hersbach Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems , 2000 .

[11]  Roberto Buizza,et al.  The Impact of Horizontal Resolution and Ensemble Size on Probabilistic Forecasts of Precipitation by the ECMWF Ensemble Prediction System , 2002 .

[12]  Leonard A. Smith,et al.  Combining dynamical and statistical ensembles , 2003 .

[13]  Thomas M. Hamill,et al.  Ensemble Reforecasting: Improving Medium-Range Forecast Skill Using Retrospective Forecasts , 2004 .

[14]  A. Raftery,et al.  Using Bayesian Model Averaging to Calibrate Forecast Ensembles , 2005 .

[15]  Anton H. Westveld,et al.  Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation , 2005 .

[16]  Martyn P. Clark,et al.  Multi‐objective calibration of forecast ensembles using Bayesian model averaging , 2006 .

[17]  Rainer Winkelmann,et al.  Analysis of Microdata , 2006 .

[18]  T. Gneiting,et al.  The continuous ranked probability score for circular variables and its application to mesoscale forecast ensemble verification , 2006 .

[19]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[20]  Thomas M. Hamill,et al.  Comparison of Ensemble-MOS Methods Using GFS Reforecasts , 2007 .

[21]  Stephen M. Stigler,et al.  c ○ Institute of Mathematical Statistics, 2007 The Epic Story of Maximum Likelihood , 2022 .

[22]  Leonard A. Smith,et al.  Increasing the Reliability of Reliability Diagrams , 2007 .

[23]  Renate Hagedorn,et al.  Probabilistic Forecast Calibration Using ECMWF and GFS Ensemble Reforecasts. Part I: Two-Meter Temperatures , 2008 .

[24]  Daniel S. Wilks,et al.  Extending logistic regression to provide full‐probability‐distribution MOS forecasts , 2009 .

[25]  Tilmann Gneiting,et al.  Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression , 2010 .

[26]  Stilian A. Stoev,et al.  CRPS M-estimation for max-stable models , 2013, 1307.7209.

[27]  M. Scheuerer Probabilistic quantitative precipitation forecasting using Ensemble Model Output Statistics , 2013, 1302.0893.

[28]  Achim Zeileis,et al.  Heteroscedastic Extended Logistic Regression for Postprocessing of Ensemble Guidance , 2014 .

[29]  Achim Zeileis,et al.  Extending Extended Logistic Regression: Extended versus Separate versus Ordered versus Censored , 2014 .

[30]  M. Scheuerer,et al.  Spatially adaptive post‐processing of ensemble forecasts for temperature , 2013, 1302.0883.

[31]  T. Hamill,et al.  Statistical Post-Processing of Ensemble Precipitation Forecasts by Fitting , 2015 .

[32]  Peter Bauer,et al.  The quiet revolution of numerical weather prediction , 2015, Nature.

[33]  Michael Scheuerer,et al.  Probabilistic wind speed forecasting on a grid based on ensemble model output statistics , 2015, 1511.02001.

[34]  Annette Moller,et al.  Probabilistic temperature forecasting based on an ensemble autoregressive modification , 2015, 1508.01397.

[35]  T. Thorarinsdottir,et al.  Spatial Postprocessing of Ensemble Forecasts for Temperature Using Nonhomogeneous Gaussian Regression , 2014, 1407.0058.

[36]  Thomas M. Hamill,et al.  Statistical Postprocessing of Ensemble Precipitation Forecasts by Fitting Censored, Shifted Gamma Distributions* , 2015 .

[37]  Nadja Klein,et al.  Bayesian structured additive distributional regression with an application to regional income inequality in Germany , 2015, 1509.05230.

[38]  Morteza Rahmani,et al.  Optimization of continuous ranked probability score using PSO , 2015 .

[39]  Christoph Eder Missing men: World War II casualties and structural change , 2016 .

[40]  Achim Zeileis,et al.  Score-Based Tests of Differential Item Functioning in the Two-Parameter Model , 2016 .

[41]  Achim Zeileis,et al.  Spatial ensemble post‐processing with standardized anomalies , 2016 .

[42]  Gottfried Tappeiner,et al.  Do methodical traps lead to wrong development strategies for welfare? A multilevel approach considering heterogeneity across industrialized and developing countries , 2016 .

[43]  S. Lang,et al.  Estimation of Spatially Correlated Random Scaling Factors based on Markov Random Field Priors , 2016 .

[44]  Olivier Mestre,et al.  Calibrated Ensemble Forecasts Using Quantile Regression Forests and Ensemble Model Output Statistics , 2016 .

[45]  Matthias Sutter,et al.  Affirmative action or just discrimination? A study on the endogenous emergence of quotas , 2016 .

[46]  Florian Pappenberger,et al.  Discrete Postprocessing of Total Cloud Cover Ensemble Forecasts , 2016 .

[47]  Jakob W. Messner,et al.  Tricks for improving non-homogeneous regression for probabilistic precipitation forecasts: Perfect predictions, heavy tails, and link functions , 2016 .

[48]  E. Dutcher,et al.  Don't hate the player, hate the game: Uncovering the foundations of cheating in contests , 2016 .

[49]  M. Halla,et al.  The Long-Lasting Shadow of the Allied Occupation of Austria on its Spatial Equilibrium , 2016, SSRN Electronic Journal.

[50]  M. Walzl,et al.  Incentive Schemes, Private Information and the Double-Edged Role of Competition for Agents , 2016 .

[51]  Achim Zeileis,et al.  A Toolkit for Stability Assessment of Tree-Based Learners , 2016 .

[52]  James M. Walker,et al.  Provision of public goods: Unconditional and conditional donations from outsiders , 2016 .

[53]  Achim Zeileis,et al.  Predictive Bookmaker Consensus Model for the UEFA Euro 2016 , 2016 .

[54]  Achim Zeileis,et al.  Heteroscedastic Censored and Truncated Regression with crch , 2016, R J..

[55]  M. Sutter,et al.  How Uncertainty and Ambiguity in Tournaments Affect Gender Differences in Competitive Behavior , 2017, European Economic Review.

[56]  L. Raynaud,et al.  The impact of horizontal resolution and ensemble size for convective‐scale probabilistic forecasts , 2017 .

[57]  Achim Zeileis,et al.  Ensemble Post-Processing of Daily Precipitation Sums over Complex Terrain Using Censored High-Resolution Standardized Anomalies , 2017 .

[58]  Duc Tran Huy,et al.  The acceptance of a protected area and the benefits of sustainable tourism: In search of the weak link in their relationship , 2017 .

[59]  Lionel Page,et al.  Can a Common Currency Foster a Shared Social Identity across Different Nations? The Case of the Euro , 2017 .

[60]  M. Geiger,et al.  The role of correlation in two-asset games: Some experimental evidence , 2017 .

[61]  Nikolaus Umlauf,et al.  Flexible Bayesian additive joint models with an application to type 1 diabetes research , 2016, Biometrical journal. Biometrische Zeitschrift.

[62]  Simon Czermak,et al.  Incentives for Dishonesty: An Experimental Study with Internal Auditors , 2017, Economic Inquiry.

[63]  M. Halla,et al.  Parental Leave, (In)formal Childcare, and Long-Term Child Outcomes , 2017, The Journal of Human Resources.

[64]  Achim Zeileis,et al.  Spatio‐temporal precipitation climatology over complex terrain using a censored additive regression model , 2016, International journal of climatology : a journal of the Royal Meteorological Society.

[65]  Achim Zeileis,et al.  Fine-Tuning Nonhomogeneous Regression for Probabilistic Precipitation Forecasts: Unanimous Predictions, Heavy Tails, and Link Functions , 2017 .

[66]  S. Renes,et al.  Fairness views and political preferences - Evidence from a large online experiment , 2017 .

[67]  Jakob W. Messner,et al.  Simultaneous Ensemble Postprocessing for Multiple Lead Times with Standardized Anomalies , 2017 .

[68]  Susanne Pech,et al.  The effect of statutory sick-pay on workers' labor supply and subsequent health , 2017 .

[69]  U. Weitzel,et al.  Rankings and Risk-Taking in the Finance Industry , 2017, The Journal of Finance.

[70]  M. Halla,et al.  Economic Origins of Cultural Norms: The Case of Animal Husbandry and Bastardy , 2017, European Economic Review.

[71]  M. Kirchler,et al.  Cash Inflow and Trading Horizon in Asset Markets , 2017 .

[72]  Achim Zeileis,et al.  Non-homogeneous boosting for predictor selection in ensemble post-processing , 2017 .

[73]  Achim Zeileis,et al.  Probabilistic Nowcasting of Low-Visibility Procedure States at Vienna International Airport During Cold Season , 2019, Pure and Applied Geophysics.

[74]  M. Halla,et al.  The Intergenerational Causal Effect of Tax Evasion: Evidence from the Commuter Tax Allowance in Austria , 2017, Journal of the European Economic Association.

[75]  Florian Wickelmaier,et al.  Using recursive partitioning to account for parameter heterogeneity in multinomial processing tree models , 2017, Behavior Research Methods.

[76]  Nikolaus Umlauf,et al.  Nonlinear association structures in flexible Bayesian additive joint models , 2017, Statistics in medicine.

[77]  Achim Zeileis,et al.  On the Estimation of Standard Errors in Cognitive Diagnosis Models , 2018 .

[78]  Achim Zeileis,et al.  BAMLSS: Bayesian Additive Models for Location, Scale, and Shape (and Beyond) , 2018, Journal of Computational and Graphical Statistics.

[79]  Michael Kirchler,et al.  Immaterial and monetary gifts in economic transactions: evidence from the field , 2017, Experimental Economics.

[80]  Stephan Rasp,et al.  Neural networks for post-processing ensemble weather forecasts , 2018, Monthly Weather Review.

[81]  Lionel Page,et al.  Guilt averse or reciprocal? Looking at behavioral motivations in the trust game , 2018, Journal of the Economic Science Association.

[82]  Daniel Müller,et al.  The anatomy of distributional preferences with group identity , 2019, Journal of Economic Behavior & Organization.

[83]  Michael Razen,et al.  Greed: Taking a deadly sin to the lab. , 2019, Journal of Behavioral and Experimental Economics.

[84]  Rudolf Kerschbamer,et al.  Do altruists lie less? , 2017, Journal of Economic Behavior & Organization.

[85]  Helena Fornwagner Incentives to lose revisited: The NHL and its tournament incentives , 2019 .

[86]  M. Geiger,et al.  Correlation and coordination risk , 2019, Annals of Finance.

[87]  S. Lang,et al.  Random scaling factors in Bayesian distributional regression models with an application to real estate data , 2019, Statistical Modelling.

[88]  Matthias Sutter,et al.  Gossip and the Efficiency of Interactions , 2019, Games Econ. Behav..

[89]  Alexander Jordan,et al.  Evaluating Probabilistic Forecasts with scoringRules , 2017, Journal of Statistical Software.

[90]  Achim Zeileis,et al.  Various versatile variances : An object-oriented implementation of clustered covariances in R Working , 2017 .

[91]  R. Kerschbamer,et al.  Social preferences and political attitudes: An online experiment on a large heterogeneous sample , 2020 .