On the Selection of Error Measures for Comparisons Among Forecasting Methods

Clements and Hendry (1993) proposed the Generalized Forecast Error Second Moment (GFESM) as an improvement to the Mean Square Error in comparing forecasting performance across data series. They based their conclusion on the fact that rankings based on GFESM remain unaltered if the series are linearly transformed. In this paper, we argue that this evaluation ignores other important criteria. Also, their conclusions were illustrated by a simulation study whose relationship to real data was not obvious. Thirdly, prior empirical studies show that the mean square error is an inappropriate measure to serve as a basis for comparison. This undermines the claims made for the GFESM.

[1]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[2]  Robert Fildes,et al.  The evaluation of extrapolative forecasting methods , 1992 .

[3]  R. Fildes,et al.  Forecasting and loss functions , 1988 .

[4]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[5]  Arnold Zellner,et al.  A tale of forecasting 1001 series : The Bayesian knight strikes again , 1986 .

[6]  Paul Newbold,et al.  ARIMA model building and the time series analysis approach to forecasting , 1983 .

[7]  Clive W. J. Granger,et al.  On the limitations of comparing mean square forecast errors: Comment , 1993 .

[8]  John Sharp,et al.  A comparison of the performance of different univariate forecasting methods in a model of capacity acquisition in UK electricity supply , 1986 .

[9]  Spyros Makridakis,et al.  Confidence intervals: An empirical investigation of the series in the M-competition , 1987 .

[10]  Patrick A. Thompson,et al.  An MSE statistic for comparing forecast accuracy across series , 1990 .

[11]  J. Scott Armstrong,et al.  Evaluation of Extrapolative Forecasting Methods: Results of a Survey of Academicians and Practitioners , 1982 .

[12]  C. Chatfield,et al.  Apples, oranges and mean square error , 1988 .

[13]  Michael P. Clements,et al.  On the limitations of comparing mean square forecast errors , 1993 .

[14]  A. H. Murphy,et al.  Diagnostic verification of probability forecasts , 1992 .

[15]  Jr. Everette S. Gardner,et al.  Evaluating forecast performance in an inventory control system , 1990 .