Benchmarking of Classical and Machine-Learning Algorithms (with special emphasis on Bagging and Boosting Approaches) for Time Series Forecasting

The goal of this Master thesis is to evaluate the time series forecast capability of several Machine Learning approaches, in detail Neural Nets, Random Forests, Kernel Machines (Support Vector Machines and Gaussian Processes), tree-based and component-wise linear and spline-based Boosting, by a comparison with classical ARIMA and ETS models. For the classical models also timeseries specific bagging approaches, Moving Block Bootstrap and Maximum Entropy Bootstrap, are tested. For this purpose, extensive benchmarks are conducted, utilizing the well-known official Tourism, M3 and NN5 competition data with the latter comprising also several exogenous covariate effects. In order to uncover specific problems the Machine Learning approaches reveal for the typical time series components of trend and seasonality, a simulation is executed helping in understanding some benchmark results as well as suggesting combinations of the Machine Learning algorithms with classical deseasoning and detrending steps (Box-Cox transformation, STL decomposition, seasonal Differencing). Furthermore different multi-step-ahead forecasting strategies are applied to the NN5 time series. It can be shown that ARIMA based models are competitive to Machine Learning models for the investigated classical (without any exogenous covariates) time series forecasting situation. On the other hand, ETS approaches are less promising. And the classical models can be enhanced by the tested bagging approaches with the easy-to-use Maximum Entropy Bootstrap showing some advantages over the more known Moving Block Bootstrap. One simple but very important result from the conducted simulation using phenotypic time series is represented by the fact that tree-based models as well as splines with a locality property are incapable of modeling a future trend. In such situations these approaches must be combined with a detrending step resulting in inferior results also for the tree-based boosting model (gbm) as one of the most popular Machine Learning algorithm. Actually the Support Vector Machine is the most promising candidate mostly outperforming all other methods including classical approaches, especially in conjunction with exogenous covariates. Even though Gaussian Processes can be founded in the same theoretical context of Kernel machines, this approach is demystified by its results. Further enhancing the Machine Learning models by direct and hybrid (combination of recursive and direct) forecasting strategies does not reveal any substantial improvements for the tested NN5 series. Interestingly, the benchmark on the M3 data conducted in this thesis seem to be the first one revealing a more than competitive prediction of special naïve methods for most series making performance conclusions of other studies based on this data highly questionable.