Visualising forecasting algorithm performance using time series instance spaces

It is common practice to evaluate the strength of forecasting methods using collections of well-studied time series datasets, such as the M3 data. The question is, though, how diverse and challenging are these time series, and do they enable us to study the unique strengths and weaknesses of different forecasting methods? This paper proposes a visualisation method for collections of time series that enables a time series to be represented as a point in a two-dimensional instance space. The effectiveness of different forecasting methods across this space is easy to visualise, and the diversity of the time series in an existing collection can be assessed. Noting that the diversity of the M3 dataset has been questioned, this paper also proposes a method for generating new time series with controllable characteristics in order to fill in and spread out the instance space, making our generalisations of forecasting method performances as robust as possible.

[1]  Xiaozhe Wang,et al.  Characteristic-Based Clustering for Time Series Data , 2006, Data Mining and Knowledge Discovery.

[2]  Kate Smith-Miles,et al.  Detecting and Classifying Events in Noisy Time Series , 2014 .

[3]  Rob J. Hyndman,et al.  Large-Scale Unusual Time Series Detection , 2015, 2015 IEEE International Conference on Data Mining Workshop (ICDMW).

[4]  Max A. Little,et al.  Highly comparative time-series analysis: the empirical structure of time series and their methods , 2013, Journal of The Royal Society Interface.

[5]  David H. Wolpert,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996, Neural Computation.

[6]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[7]  Jeffrey S. Racine,et al.  Entropy and predictability of stock market returns , 2002 .

[8]  Yannis Manolopoulos,et al.  Feature-based classification of time-series data , 2001 .

[9]  Nick S. Jones,et al.  Highly Comparative Feature-Based Time-Series Classification , 2014, IEEE Transactions on Knowledge and Data Engineering.

[10]  Víctor Leiva,et al.  An R Package for a General Class of Inverse Gaussian Distributions , 2008 .

[11]  Giovanni Zurlini,et al.  Order and disorder in ecological time-series: Introducing normalized spectral entropy , 2013 .

[12]  Rob J Hyndman,et al.  A state space framework for automatic forecasting using exponential smoothing methods , 2002 .

[13]  Badong Chen,et al.  Weighted-permutation entropy: a complexity measure for time series incorporating amplitude information. , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  B. Pompe,et al.  Permutation entropy: a natural complexity measure for time series. , 2002, Physical review letters.

[15]  George C. Runger,et al.  A time series forest for classification and feature extraction , 2013, Inf. Sci..

[16]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[17]  Luca Scrucca,et al.  GA: A Package for Genetic Algorithms in R , 2013 .

[18]  K. Nikolopoulos,et al.  The theta model: a decomposition approach to forecasting , 2000 .

[19]  Kate Smith-Miles,et al.  Towards objective measures of algorithm performance across instance space , 2014, Comput. Oper. Res..

[20]  Georg M. Goerg.  Forecastable Component Analysis , 2013, ICML.

[21]  George Athanasopoulos,et al.  Forecasting: principles and practice , 2013 .

[22]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[23]  Joshua Garland,et al.  Model-free quantification of time-series predictability. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[24]  Fotios Petropoulos,et al.  forecast: Forecasting functions for time series and linear models , 2018 .

[25]  Fotios Petropoulos,et al.  'Horses for Courses' in demand forecasting , 2014, Eur. J. Oper. Res..

[26]  Kate Smith-Miles,et al.  Generating new test instances by evolving in instance space , 2015, Comput. Oper. Res..

[27]  B. Silver,et al.  Order and Disorder , 1985 .

[28]  K. Smith‐Miles,et al.  Classes of structures in the stable atmospheric boundary layer , 2015 .

[29]  T. Sejnowski,et al.  Quantifying neighbourhood preservation in topographic mappings , 1996 .

[30]  Spyros Makridakis,et al.  The M3-Competition: results, conclusions and implications , 2000 .

[31]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[32]  Irma J. Terpenning,et al.  STL : A Seasonal-Trend Decomposition Procedure Based on Loess , 1990 .