Forecasting in social settings: The state of the art

Abstract This paper provides a non-systematic review of the progress of forecasting in social settings. It is aimed at someone outside the field of forecasting who wants to understand and appreciate the results of the M4 Competition, and forms a survey paper regarding the state of the art of this discipline. It discusses the recorded improvements in forecast accuracy over time, the need to capture forecast uncertainty, and things that can go wrong with predictions. Subsequently, the review classifies the knowledge achieved over recent years into (i) what we know, (ii) what we are not sure about, and (iii) what we don’t knowIn the first two areas, we explore the difference between explanation and prediction, the existence of an optimal model, the performance of machine learning methods on time series forecasting tasks, the difficulties of predicting non-stable environments, the performance of judgment, and the value added by exogenous variables. The article concludes with the importance of (thin and) fat tails, the challenges and advances in causal inference, and the role of luck.

[1]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[2]  George Athanasopoulos,et al.  Forecasting: principles and practice , 2013 .

[3]  Fotios Petropoulos,et al.  Judgmental selection of forecasting models , 2018 .

[4]  K. Torkkola,et al.  A Multi-Horizon Quantile Recurrent Forecaster , 2017, 1711.11053.

[5]  C. Chatfield,et al.  The M2-competition: A real-time judgmentally based forecasting study , 1993 .

[6]  Philip Hans Franses.  A simple test for a bubble based on growth and acceleration , 2016, Comput. Stat. Data Anal..

[7]  Herman Stekler,et al.  Evaluating qualitative forecasts: The FOMC minutes, 2006–2010 , 2016 .

[8]  Robert C. Blattberg,et al.  Database Models And Managerial Intuition: 50% Model + 50% Manager , 1990 .

[9]  Marcus O'Connor,et al.  An examination of the accuracy of judgmental extrapolation of time series , 1985 .

[10]  Sven F. Crone,et al.  Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction , 2011 .

[11]  Peter J. Danaher,et al.  Econometric models for forecasting market share , 2001 .

[12]  William Remus,et al.  Judgemental forecasting in times of change , 1993 .

[13]  Richard A. Ashley,et al.  On the relative worth of recent macroeconomic forecasts , 1988 .

[14]  James Surowiecki.  The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[15]  R. Brown,et al.  Smoothing, Forecasting, and Prediction of Discrete Time Series , 1965 .

[16]  Rob J. Hyndman,et al.  FFORMA: Feature-based forecast model averaging , 2020, International Journal of Forecasting.

[17]  Fotios Petropoulos,et al.  'Horses for Courses' in demand forecasting , 2014, Eur. J. Oper. Res..

[18]  Alexander M. Millkey.  The Black Swan: The Impact of the Highly Improbable , 2009 .

[19]  M. Iredell,et al.  The NCEP Climate Forecast System Version 2 , 2014 .

[20]  Paul Goodwin.  Forewarned: A Sceptic's Guide to Prediction , 2017 .

[21]  Spyros Makridakis,et al.  The M3-Competition: results, conclusions and implications , 2000 .

[22]  Fotios Petropoulos,et al.  Using a rolling training approach to improve judgmental extrapolations elicited from forecasters with technical knowledge , 2017 .

[23]  J. Armstrong,et al.  Findings from Evidence-Based Forecasting: Methods for Reducing Forecast Error , 2006 .

[24]  Evangelos Spiliotis,et al.  Statistical and Machine Learning forecasting methods: Concerns and ways forward , 2018, PloS one.

[25]  R. Fildes,et al.  The State of Macroeconomic Forecasting , 2002 .

[26]  A. Timmermann.  Forecast Combinations , 2005 .

[27]  R. Fildes,et al.  Analysis of judgmental adjustments in the presence of promotions , 2013 .

[28]  Rob J Hyndman,et al.  Automatic Time Series Forecasting: The forecast Package for R , 2008 .

[29]  Steven P. Schnaars,et al.  Megamistakes: Forecasting and the Myth of Rapid Technological Change, Steven P. Schnaars. 1988. Free Press. 224 pages. ISBN: 0-02-927952-6. $19.95 , 1989 .

[30]  T. Evgeniou,et al.  To combine or not to combine: selecting among forecasts and their combinations , 2005 .

[31]  W. Collins,et al.  The NCEP–NCAR 50-Year Reanalysis: Monthly Means CD-ROM and Documentation , 2001 .

[32]  Spyros Makridakis,et al.  The art and science of forecasting An assessment and future directions , 1986 .

[33]  Leonard J. Tashman,et al.  Out-of-sample tests of forecasting accuracy: an analysis and review , 2000 .

[34]  Rob J Hyndman,et al.  Bagging exponential smoothing methods using STL decomposition and Box–Cox transformation , 2016 .

[35]  Michèle Hibon,et al.  Accuracy of Forecasting: An Empirical Investigation , 1979 .

[36]  Philip Hans Franses,et al.  Properties of expert adjustments on model-based SKU-level forecasts , 2009 .

[37]  Xiaozhe Wang,et al.  Rule induction for forecasting method selection: Meta-learning the characteristics of univariate time series , 2009, Neurocomputing.

[38]  Evangelos Spiliotis,et al.  The M4 Competition: Results, findings, conclusion and way forward , 2018, International Journal of Forecasting.

[39]  Nada R. Sanders,et al.  Accuracy of judgmental forecasts: A comparison , 1992 .

[40]  Spyros Makridakis,et al.  Decision making and planning under low levels of predictability , 2009 .

[41]  Amir F. Atiya,et al.  Combination of long term and short term forecasts, with application to tourism demand forecasting , 2011 .

[42]  H. Stekler,et al.  Has The Accuracy of Macroeconomic Forecasts for Germany Improved? , 2013 .

[43]  C. Granger.  Investigating causal relations by econometric models and cross-spectral methods , 1969 .

[44]  Konstantinos Nikolopoulos,et al.  The Tourism Forecasting Competition , 2011 .

[45]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[46]  Hirotugu Akaike,et al.  On entropy maximization principle , 1977 .

[47]  J. Pearl.  Causal diagrams for empirical research , 1995 .

[48]  Fotios Petropoulos,et al.  Forecasting with temporal hierarchies , 2017, Eur. J. Oper. Res..

[49]  Judea Pearl,et al.  Identification of Conditional Interventional Distributions , 2006, UAI.

[50]  Philip Hans Franses,et al.  Combining SKU-level sales forecasts from models and experts , 2009, Expert Syst. Appl..

[51]  K. Nikolopoulos,et al.  The theta model: a decomposition approach to forecasting , 2000 .

[52]  Fotios Petropoulos,et al.  forecast: Forecasting functions for time series and linear models , 2018 .

[53]  Spyros Makridakis,et al.  Why combining works , 1989 .

[54]  Jun Wang,et al.  Forecasting stochastic neural network based on financial empirical mode decomposition , 2017, Neural Networks.

[55]  Miguel A. Vadillo,et al.  Illusion of Control , 2013, Experimental psychology.

[56]  P. Tetlock.  Expert Political Judgment: How Good Is It? How Can We Know? , 2005 .

[57]  Amir F. Atiya,et al.  An Empirical Comparison of Machine Learning Models for Time Series Forecasting , 2010 .

[58]  Fotios Petropoulos,et al.  To select or to combine? The inventory performance of model and expert forecasts , 2016 .

[59]  Fred Collopy,et al.  Automatic Identification of Time Series Features for Rule-Based Forecasting , 2001 .

[60]  Philip Hans Franses,et al.  Do experts' adjustments on model-based SKU-level forecasts improve forecast quality? , 2009 .

[61]  G. Gaus,et al.  Expert Political Judgment: How Good Is It? How Can We Know? , 2007, Perspectives on Politics.

[62]  Evangelos Spiliotis,et al.  The M4 Competition: 100,000 time series and 61 forecasting methods , 2020 .

[63]  Wilpen L. Gorr,et al.  ACCURACY OF JUDGMENTAL FORECASTING OF TIME SERIES , 1985 .

[64]  F. Diebold,et al.  On the Comparison of Interval Forecasts , 2018, Journal of Time Series Analysis.

[65]  J. Pearl.  Causality: Models, Reasoning and Inference , 2000 .

[66]  Robert Fildes,et al.  Evaluation of Aggregate and Individual Forecast Method Selection Rules , 1989 .

[67]  Fotios Petropoulos,et al.  An evaluation of simple versus complex selection rules for forecasting many time series , 2014 .

[68]  Salim Yusuf,et al.  Evolving evidence about diet and health. , 2018, The Lancet. Public health.

[69]  S. Kolassa.  Combining exponential smoothing forecasts using Akaike weights , 2011 .

[70]  Michael Scott,et al.  Delphi: A History of the Center of the Ancient World , 2014 .

[71]  William H. Starbuck,et al.  Innocents in the Forest: Forecasting and Research Methods , 1990 .

[72]  Jim Burton,et al.  Robert FitzRoy and the Early History of the Meteorological Office , 1986, The British Journal for the History of Science.

[73]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[74]  Martin A. Wildy.  Interpreting Financial Market Crashes as Earthquakes: A New Early Warning System for Medium Term Crashes , 2015 .

[75]  M. Fitzgerald,et al.  Horses for courses. , 2004, International journal of nursing practice.

[76]  R. Brown.  Statistical forecasting for inventory control , 1960 .

[77]  Herman Stekler,et al.  The future of macroeconomic forecasting: Understanding the forecasting process , 2007 .

[78]  Dariush Mozaffarian,et al.  Red and Processed Meat Consumption and Risk of Incident Coronary Heart Disease, Stroke, and Diabetes Mellitus: A Systematic Review and Meta-Analysis , 2010, Circulation.

[79]  R. L. Winkler.  A Decision-Theoretic Approach to Interval Estimation , 1972 .

[80]  R. Cooke,et al.  Fat-Tailed Distributions: Data, Diagnostics and Dependence , 2014 .

[81]  P. Goodwin,et al.  Judgmental forecasting: A review of progress over the last 25 years , 2006 .

[82]  Nigel Harvey,et al.  Why Are Judgments Less Consistent in Less Predictable Task Situations , 1995 .

[83]  F. Petropoulos,et al.  Improving forecasting by estimating time series structural components across multiple frequencies , 2014 .

[84]  J. Pearl,et al.  The Book of Why: The New Science of Cause and Effect , 2018 .

[85]  Rob J Hyndman,et al.  Another look at measures of forecast accuracy , 2006 .

[86]  Fotios Petropoulos,et al.  Exploring the sources of uncertainty: Why does bagging for time series forecasting work? , 2018, Eur. J. Oper. Res..

[87]  F. Bolger,et al.  Graphs versus tables: Effects of data presentation format on judgemental forecasting , 1996 .

[88]  Rob J Hyndman,et al.  A state space framework for automatic forecasting using exponential smoothing methods , 2002 .

[89]  Pierre Pinson,et al.  Global Energy Forecasting Competition 2012 , 2014 .

[90]  Jerry G. Thursby,et al.  Replication in Empirical Economics: The Journal of Money, Credit and Banking Project , 1986 .

[91]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[92]  George Wright,et al.  Use of expert knowledge to anticipate the future: Issues, analysis and directions , 2017 .

[93]  Tao Hong,et al.  Global energy forecasting competition 2017: Hierarchical probabilistic load forecasting , 2019, International Journal of Forecasting.

[94]  Judea Pearl,et al.  Comment: Graphical Models, Causality and Intervention , 2016 .

[95]  Souhaib Ben Taieb,et al.  A gradient boosting approach to the Kaggle load forecasting competition , 2014 .

[96]  Philip E. Tetlock,et al.  Superforecasting: The Art and Science of Prediction , 2015 .

[97]  Robert Fildes,et al.  Demand forecasting with high dimensional data: The case of SKU retail sales forecasting with intra- and inter-category promotional information , 2016, Eur. J. Oper. Res..

[98]  Nigel Harvey,et al.  Sensitivity to autocorrelation in judgmental time series forecasting , 2011 .

[99]  Fotios Petropoulos,et al.  Another look at forecast selection and combination: Evidence from forecast pooling , 2019, International Journal of Production Economics.

[100]  R. Fildes,et al.  Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning , 2009 .

[101]  Spyros Makridakis,et al.  Factors affecting judgmental forecasts and confidence intervals , 1989 .

[102]  Rob J Hyndman,et al.  Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond , 2016 .

[103]  J. Brooks.  Why most published research findings are false: Ioannidis JP, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece , 2008 .

[104]  Tobias F. Rötheli.  Superforecasting: the art and science of prediction , 2017 .

[105]  Marcus O'Connor,et al.  Forecasting support systems for the incorporation of event information: An empirical investigation , 2005 .

[106]  Valentin Flunkert,et al.  DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks , 2017, International Journal of Forecasting.