A new accuracy measure based on bounded relative error for time series forecasting

Many accuracy measures have been proposed in the past for time series forecasting comparisons. However, many of these measures suffer from one or more issues such as poor resistance to outliers and scale dependence. In this paper, while summarising commonly used accuracy measures, a special review is made on the symmetric mean absolute percentage error. Moreover, a new accuracy measure called the Unscaled Mean Bounded Relative Absolute Error (UMBRAE), which combines the best features of various alternative measures, is proposed to address the common issues of existing measures. A comparative evaluation on the proposed and related measures has been made with both synthetic and real-world data. The results indicate that the proposed measure, with user selectable benchmark, performs as well as or better than other measures on selected criteria. Though it has been commonly accepted that there is no single best accuracy measure, we suggest that UMBRAE could be a good choice to evaluate forecasting methods, especially for cases where measures based on geometric mean of relative errors, such as the geometric mean relative absolute error, are preferred.

[1]  Michael Small,et al.  Complex network analysis of time series , 2016 .

[2]  Wei-Dong Dang,et al.  Multiscale limited penetrable horizontal visibility graph for analyzing nonlinear time series , 2016, Scientific Reports.

[3]  Zhong-Ke Gao,et al.  Multiscale complex network for analyzing experimental multivariate time series , 2015 .

[4]  Zaccheus O. Olaofe,et al.  A 5-day wind speed & power forecasts using a layer recurrent neural network (LRNN) , 2014 .

[5]  Vjekoslav Galzina,et al.  An adaptive network-based fuzzy inference system (ANFIS) for the forecasting: The case of close price indices , 2013, Expert Syst. Appl..

[6]  R. Fildes,et al.  Measuring forecasting accuracy : the case of judgmental adjustments to SKU-level demand forecasts , 2013 .

[7]  Zhongke Gao,et al.  A directed weighted complex network for characterizing chaotic dynamics from time series , 2012 .

[8]  Tai-Liang Chen,et al.  A hybrid model based on adaptive-network-based fuzzy inference system to forecast Taiwan stock market , 2011, Expert Syst. Appl..

[9]  Derya Avci,et al.  An Adaptive Network-Based Fuzzy Inference System (ANFIS) for the prediction of stock market return: The case of the Istanbul Stock Exchange , 2010, Expert Syst. Appl..

[10]  Akbar Esfahanipour,et al.  Adapted Neuro-Fuzzy Inference System on indirect approach TSK fuzzy rule base for stock market analysis , 2010, Expert Syst. Appl..

[11]  George Bernard Shaw,et al.  LONG-RANGE FORECASTING From Crystal Ball to Computer , 2010 .

[12]  H. Simon.  LONG-RANGE FORECASTING From Crystal Ball to Computer , 2010 .

[13]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[14]  Rob J Hyndman,et al.  25 years of time series forecasting , 2006 .

[15]  Yuhong Yang,et al.  Assessing Forecast Accuracy Measures , 2004 .

[16]  J. Armstrong,et al.  Evaluating Forecasting Methods , 2001 .

[17]  Spyros Makridakis,et al.  The M3-Competition: results, conclusions and implications , 2000 .

[18]  P. Goodwin,et al.  On the asymmetry of the symmetric MAPE , 1999 .

[19]  J. Scott Armstrong,et al.  On the Selection of Error Measures for Comparisons Among Forecasting Methods , 2005 .

[20]  Michael P. Clements,et al.  On the limitations of comparing mean square forecast errors , 1993 .

[21]  Spyros Makridakis,et al.  Accuracy measures: theoretical and practical concerns☆ , 1993 .

[22]  Robert Fildes,et al.  The evaluation of extrapolative forecasting methods , 1992 .

[23]  Fred L. Collopy,et al.  Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons , 1992 .

[24]  C. Chatfield,et al.  Apples, oranges and mean square error , 1988 .

[25]  Philip J. Fleming,et al.  How not to lie with statistics: the correct way to summarize benchmark results , 1986, CACM.

[26]  G. Capon,et al.  Evaluation of forecasting methods for decision support , 1986 .

[27]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[28]  Steven C. Wheelwright,et al.  Forecasting methods and applications. , 1979 .