Evaluating Forecasting Methods by Considering Different Accuracy Measures

Abstract Choosing the appropriate forecasting technique to employ is a challenging issue and requires a comprehensive analysis of empirical results. Recent research findings reveal that the performance evaluation of forecasting models depends on the accuracy measures adopted. Some methods indicate superior performance when error based metrics are used, while others perform better when precision values are adopted as accuracy measures. As scholars tend to use a smaller subset of accuracy metrics to assess the performance of forecasting models, there is a need for a concept of multiple accuracy dimensions to assure the robustness of evaluation. Therefore, the main purpose of this paper is to propose a decision making model that allows researchers to identify the superiority of a forecasting technique over another by considering several accuracy metrics concurrently. A multi-criteria decision analysis approach, namely the preference ranking organization method for enrichment evaluation (PROMETHEE), was adopted to solve this problem. Bayesian Networks, Artificial Neural Networks, SVMs, Logistic Regression, and several Rule and Tree-based forecasting approaches were included in the analysis. After introducing a detailed description of accuracy measures, the performance of the prediction models are evaluated using a chosen dataset from the UCI Machine Learning Repository.