Automatic Modelling and Forecasting with Artificial Neural Networks- A forecasting competition evaluation

During 2007 we conducted an empirical evaluation of the accuracy of artificial Neural Networks (NN) and other methods of Computational Intelligence (CI) in time series prediction through a dedicated forecasting competition: the NN3 (www.neural-forecasting-competition.com). The competition aimed to resolve two research questions: (a) what is the current performance of CI methods in comparison to established statistical forecasting methods, and (b) what are the current “best practices” regarding the methodologies to model CI such as NN for time series forecasting. The NN3 competition evaluated the ex ante accuracy of multiple step ahead predictions across multiple error metrics. The data sample contained two homogeneous sets of 111 or 11 time series of varying length (short and long) and different time series patterns (seasonal and non-seasonal) taken from the original M3-competition, in order to analyse the conditions under which a particular method would perform well and to compare the accuracy to the contenders in the earlier competition. The NN3 competition attracted 60 submissions of CI methods as well as novel statistical contenders, one of the largest forecasting competitions conducted to date. The final results suggest that for monthly time series of different length and seasonality a variety of different CI methods are capable of forecasting automatically using a consistent methodology and show a robust and comparative performance, but that statistical methods still outperform the majority of CI- methods.