The use of protocols to select exponential smoothing procedures: A reconsideration of forecasting competitions

Forecasting competitions have evaluated the accuracy of extrapolative methods unselectively; that is, by applying each method to an entire batch of time series without asking whether the method is appropriate for each individual series. For example, simple exponential smoothing was applied to trended as well as untrended time series. In consequence, the calculated error measures do not unambiguously portray the accuracy one can expect in using a given procedure selectively. In this study, we employ three different protocols for selecting appropriate exponential smoothing procedures. Then, based on 132 time series from previous forecasting competitions, we compare forecasting accuracy in cases for which the procedures are designated (by each protocol) as appropriate with accuracy achieved in applications designated as inappropriate. The selection protocols are a variance analysis proposed by Gardner and McKenzie, a set of visual rules adapted from Collopy and Armstrong's rule-based forecasting system, and a method-switching procedure developed by Goodrich. We found the selection protocols to be effective in selecting appropriate applications of strong-trend (linear and exponential trend) procedures. For example, when the Holt (linear trend) procedure is applied selectively, its forecasting accuracy is much superior to that suggested by the forecasting competitions. On the other hand, the protocols are not effectively distinguishing appropriate from inappropriate applications of damped and no-trend procedures. Further, our results reveal considerable disagreement among the protocols themselves, making it apparent that much work remains to be done to improve the state of the art in forecasting-method selection.

[1]  Steven P. Schnaars.  An Evaluation of Rules for Selecting an Extrapolation Model on Yearly Sales Forecasts , 1986 .

[2]  Andrew Zardecki,et al.  Rule-Based Forecasting , 1996 .

[3]  Everette S. Gardner,et al.  Model Identification in Exponential Smoothing , 1988 .

[4]  Peter R. Winters,et al.  Forecasting Sales by Exponentially Weighted Moving Averages , 1960 .

[5]  Fred Collopy,et al.  Rule-Based Forecasting: Development and Validation of an Expert Systems Approach to Combining Time Series Extrapolations , 1992 .

[6]  C. C. Holt,et al.  Planning Production, Inventories, and Work Force. , 1962 .

[7]  Robert Fildes,et al.  The evaluation of extrapolative forecasting methods , 1992 .

[8]  Leonard J. Tashman,et al.  Automatic forecasting software: A survey and evaluation☆ , 1991 .

[9]  J. Keith Ord,et al.  Automatic forecasting using explanatory variables: A comparative study , 1991 .

[10]  E. S. Gardner,et al.  Forecasting Trends in Time Series , 1985 .

[11]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[12]  Robert Fildes,et al.  Evaluation of Aggregate and Individual Forecast Method Selection Rules , 1989 .

[13]  Everette S. Gardner,et al.  Exponential smoothing: The state of the art , 1985 .

[14]  C. Chatfield,et al.  The M2-competition: A real-time judgmentally based forecasting study , 1993 .

[15]  G. Schwarz.  Estimating the Dimension of a Model , 1978 .

[16]  Chris Chatfield,et al.  The Holt-Winters Forecasting Procedure , 1978 .

[17]  R. Brown.  Statistical forecasting for inventory control , 1960 .

[18]  P. Newbold,et al.  Introductory Business & Economic Forecasting , 1993 .

[19]  Robert L. Winkler,et al.  Sampling Distributions of Post-sample Forecasting Errors , 1989 .

[20]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[21]  Steven P. Schnaars.  A comparison of extrapolation models on yearly sales forecasts , 1986 .

[22]  Spyros Makridakis,et al.  Sliding simulation: a new approach to time series forecasting , 1990 .

[23]  Fred Collopy,et al.  Causal Forces: Structuring Knowledge for Time-Series Extrapolation , 1993 .