Integrating human judgement into quantitative forecasting methods: A review

Abstract Product forecasts are a critical input into sourcing, procurement, production, inventory, logistics, finance and marketing decisions. Numerous quantitative models have been developed and applied to generate and improve product forecasts. The use of human judgement, either solely or in conjunction with quantitative models, has been well researched in the academic literature and is a popular forecasting approach in industry practice. In the context of judgemental forecasting, methods that integrate an expert's judgement into quantitative forecasting models are commonly referred to as “integrating forecasting” methods. This paper presents a systematic review of the literature of judgemental demand forecasting with a focus placed on integrating methods. We explore the role of expert opinion and contextual information and discuss the application of behaviourally informed support systems. We also provide important directions for further research in these areas.

[1]  Konstantinos Nikolopoulos,et al.  Forecasting with quantitative methods: the impact of special events in time series , 2010 .

[2]  Robert Fildes,et al.  Against Your Better Judgment? How Organizations Can Improve Their Use of Management Judgment in Forecasting , 2007, Interfaces.

[3]  Hau L. Lee,et al.  Information distortion in a supply chain: the bullwhip effect , 1997 .

[4]  Larry P. Ritzman,et al.  The need for contextual and technical knowledge in judgmental forecasting , 1992 .

[5]  Robert Fildes,et al.  Judgmental forecasts of time series affected by special events: does providing a statistical forecast improve accuracy? , 1999 .

[6]  Konstantinos Nikolopoulos,et al.  The process of using a forecasting support system , 2007 .

[7]  Vera Shanshan Lin,et al.  A review of Delphi forecasting research in tourism , 2015 .

[8]  Nada R. Sanders,et al.  The impact of task properties feedback on time series judgmental forecasting tasks , 1997 .

[9]  Stephen M. Disney,et al.  Forecasting for inventory planning: a 50-year review , 2009, J. Oper. Res. Soc..

[10]  Robert Fildes,et al.  Restrictiveness and guidance in support systems , 2011 .

[11]  Philip Hans Franses,et al.  Merging models and experts , 2008 .

[12]  Jack T. Dennerlein,et al.  Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting , 2017 .

[13]  Monica Adya,et al.  Development and validation of a rule-based time series complexity scoring technique to support design of adaptive forecasting DSS , 2016, Decis. Support Syst..

[14]  Christopher M. Miller,et al.  Applying case-based reasoning to forecasting retail sales , 1993 .

[15]  Özalp Özer,et al.  Trust in Forecast Information Sharing , 2009, Manag. Sci..

[16]  Fotios Petropoulos,et al.  Amplifying the learning effect via a forecasting and foresight support system , 2015 .

[17]  J. Scott Armstrong,et al.  Principles of forecasting : a handbook for researchers and practitioners , 2001 .

[18]  Nada R. Sanders,et al.  Forecasting Software in Practice: Use, Satisfaction, and Performance , 2003, Interfaces.

[19]  Manoj Kumar Tiwari,et al.  Big data and predictive analytics applications in supply chain management , 2016, Comput. Ind. Eng..

[20]  Paul Goodwin,et al.  Integrating management judgment and statistical methods to improve short-term forecasts , 2002 .

[21]  Marta Mas Machuca,et al.  A review of forecasting models for new products , 2014 .

[22]  Larry P. Ritzman,et al.  Judgmental Adjustment of Statistical Forecasts , 2001 .

[23]  R. Fildes,et al.  Providing support for the use of analogies in demand forecasting tasks , 2007 .

[24]  Joseph P. Simmons,et al.  Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them , 2016, Manag. Sci..

[25]  Stephen J. Hoch,et al.  A psychological approach to decision support systems , 1996 .

[26]  Keely L. Croxton,et al.  Biases in judgmental adjustments of statistical forecasts: The role of individual differences , 2010 .

[27]  R. H. Edmundson,et al.  The accuracy of combining judgemental and statistical forecasts , 1986 .

[28]  Paul Goodwin,et al.  Correct or combine? Mechanically integrating judgmental forecasts with statistical methods , 2000 .

[29]  H.L. Lee,et al.  Aligning Supply Chain Strategies with Product Uncertainties , 2002, IEEE Engineering Management Review.

[30]  Joa Sang Lim,et al.  Judgmental forecasting with interactive forecasting support systems , 1996, Decis. Support Syst..

[31]  Meredith Lawley,et al.  Factors influencing decision support system acceptance , 2013, Decis. Support Syst..

[32]  Michael Lawrence,et al.  A field study of sales forecasting accuracy and processes , 2000, Eur. J. Oper. Res..

[33]  Stavros Asimakopoulos,et al.  Forecasting support systems technologies-in-practice: A model of adoption and use for product forecasting , 2013 .

[34]  Robert L. Winkler,et al.  The accuracy of extrapolation (time series) methods: Results of a forecasting competition , 1982 .

[35]  Philip Hans Franses,et al.  Averaging Model Forecasts and Expert Forecasts: Why Does It Work? , 2011, Interfaces.

[36]  Muammer Ozer.  The Use of Internet-Based Groupware in New Product Forecasting , 1999 .

[37]  J. Scott Armstrong,et al.  Combining forecasts: The end of the beginning or the beginning of the end? , 1989 .

[38]  Fred Collopy,et al.  Automatic Identification of Time Series Features for Rule-Based Forecasting , 2001 .

[39]  Haroon Idrees,et al.  Calculating Web impact factor for university websites of Pakistan , 2015, Electron. Libr..

[40]  J. Armstrong,et al.  Findings from Evidence-Based Forecasting: Methods for Reducing Forecast Error , 2006 .

[41]  Robert Fildes,et al.  The design features of forecasting support systems and their effectiveness , 2006, Decis. Support Syst..

[42]  Marcus O'Connor,et al.  Forecasting support systems for the incorporation of event information: An empirical investigation , 2005 .

[43]  Ulrike Leopold-Wildburger,et al.  Experiments on forecasting behavior with several sources of information - A review of the literature , 2011, Eur. J. Oper. Res..

[44]  Aris A. Syntetos,et al.  Spare parts management : a review of forecasting research and extensions , 2010 .

[45]  M. Petticrew,et al.  Systematic Reviews in the Social Sciences: A Practical Guide , 2005 .

[46]  Marcus O'Connor,et al.  The effect of group interaction processes on performance in time series extrapolation , 1991 .

[47]  Nada R. Sanders,et al.  The efficacy of using judgmental versus quantitative forecasting methods in practice , 2003 .

[48]  Marcus O'Connor,et al.  The use of non‐time series information in sales forecasting: A case study , 1988 .

[49]  Marshall L. Fisher,et al.  Estimating Demand Uncertainty Using Judgmental Forecasts , 2007, Manuf. Serv. Oper. Manag..

[50]  Vera Shanshan Lin,et al.  Accuracy and bias of experts’ adjusted forecasts , 2014 .

[51]  Kesten C. Green,et al.  Golden Rule of Forecasting: Be Conservative , 2015 .

[52]  N. Sanders,et al.  Forecasting Practices in US Corporations: Survey Results , 1994 .

[53]  Nigel Harvey,et al.  Use of heuristics: Insights from forecasting research , 2007, Thinking & Reasoning.

[54]  Bert de Bruijn,et al.  Heterogeneous Forecast Adjustment , 2016 .

[55]  J. Armstrong,et al.  The Ombudsman: Research on Forecasting: A Quarter-Century Review, 1960--1984 , 1986 .

[56]  P. Goodwin,et al.  Judgmental forecasting: A review of progress over the last 25 years , 2006 .

[57]  Nigel Harvey,et al.  Why Are Judgments Less Consistent in Less Predictable Task Situations , 1995 .

[58]  Philip Hans Franses,et al.  Experts' adjustment to model-based SKU-level forecasts: does the forecast horizon matter? , 2011, J. Oper. Res. Soc..

[59]  E. Stanley Lee,et al.  A forecasting decision on the sales volume of printers in Taiwan: An exploitation of the Analytic Network Process , 2012, Comput. Math. Appl..

[60]  Dominique M. Hanssens,et al.  Do Promotions Benefit Manufacturers, Retailers, or Both? , 2002, Manag. Sci..

[61]  Paul Goodwin,et al.  Providing support for decisions based on time series information under conditions of asymmetric loss , 2005, Eur. J. Oper. Res..

[62]  Valerie Belton,et al.  Remarks on the application of the analytic hierarchy process to judgmental forecasting , 1996 .

[63]  Derek W. Bunn,et al.  Non-traditional methods of forecasting , 1996 .

[64]  James S. Dyer,et al.  The Reliability of Subjective Probabilities: Obtained Through Decomposition , 2015 .

[65]  Robert Fildes,et al.  Principles of Business Forecasting , 2012 .

[66]  Petros Ieromonachou,et al.  Big data analytics in supply chain management: A state-of-the-art literature review , 2017, Comput. Oper. Res..

[67]  Ching-Chin Chern,et al.  Designing a decision-support system for new product sales forecasting , 2010, Expert Syst. Appl..

[68]  Janet A. Sniezek,et al.  An examination of group process in judgmental forecasting , 1989 .

[69]  Enno Siemsen,et al.  Judgmental Forecasting: Cognitive Reflection and Decision Speed , 2014 .

[70]  Robert C. Blattberg,et al.  Database Models And Managerial Intuition: 50% Model + 50% Manager , 1990 .

[71]  Spyros Makridakis,et al.  Forecasting and Planning: An Evaluation , 1981 .

[72]  William Remus,et al.  Judgemental forecasting in times of change , 1993 .

[73]  Nigel Harvey,et al.  Effects of judges' forecasting on their later combination of forecasts for the same outcomes , 2004 .

[74]  Muammer Ozer,et al.  Understanding the impacts of product knowledge and product type on the accuracy of intentions-based new product predictions , 2011, Eur. J. Oper. Res..

[75]  Stefano Ronchi,et al.  Integrating quantitative and qualitative forecasting approaches: organizational learning in an action research case , 2011, J. Oper. Res. Soc..

[76]  Fotios Petropoulos,et al.  Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour? , 2016 .

[77]  D. MacGregor.  Decomposition for Judgmental Forecasting and Estimation , 1999 .

[78]  Haiyan Song,et al.  Developing a Web-Based Tourism Demand Forecasting System , 2008 .

[79]  Phillip M. Yelland,et al.  A Bayesian Model for Sales Forecasting at Sun Microsystems , 2010, Interfaces.

[80]  Fred Collopy,et al.  Rule-Based Forecasting: Development and Validation of an Expert Systems Approach to Combining Time Series Extrapolations , 1992 .

[81]  R. Fildes,et al.  Analysis of judgmental adjustments in the presence of promotions , 2013 .

[82]  Robert C. Blattberg,et al.  Sales Promotion: Concepts, Methods, and Strategies , 1990 .

[83]  R. Fildes,et al.  Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning , 2009 .

[84]  Naoufel Cheikhrouhou,et al.  Structuring and integrating human knowledge in demand forecasting: a judgemental adjustment approach , 2010 .

[85]  Larry P. Ritzman,et al.  Bringing judgment into combination forecasts , 1995 .

[86]  R. H. Edmundson.  Decomposition; a strategy for judgemental forecasting , 1990 .

[87]  Enno Siemsen,et al.  Demand Forecasting Behavior: System Neglect and Change Detection , 2011, Manag. Sci..

[88]  P. Hofer,et al.  Forecasting in Austrian companies , 2015 .

[89]  M. O'Connor,et al.  Judgemental and statistical time series forecasting: a review of the literature , 1996 .

[90]  D. Bunn,et al.  Interaction of judgemental and statistical forecasting methods: issues & , 1991 .

[91]  P. Goodwin.  Improving the voluntary integration of statistical forecasts and judgment , 2000 .

[92]  Paul Goodwin,et al.  Heuristics, biases and improvement strategies in judgmental time series forecasting , 1994 .

[93]  Paul P. Wang,et al.  Intelligent system to support judgmental business forecasting: the case of estimating hotel room demand , 2000, IEEE Trans. Fuzzy Syst..

[94]  Jorge A. Alvarado-Valencia,et al.  Reliance, trust and heuristics in judgmental forecasting , 2014, Comput. Hum. Behav..

[95]  Naoufel Cheikhrouhou,et al.  A collaborative demand forecasting process with event-based fuzzy judgements , 2011, Comput. Ind. Eng..

[96]  Nada R. Sanders,et al.  Accuracy of judgmental forecasts: A comparison , 1992 .

[97]  J. Carbonell.  Learning by Analogy: Formulating and Generalizing Plans from Past Experience , 1983 .

[98]  Robert Carbone,et al.  Comparing for Different Time Series Methods the Value of Technical Expertise Individualized Analysis, and Judgmental Adjustment , 1983 .

[99]  Juan R. Trapero,et al.  On the identification of sales forecasting models in the presence of promotions , 2015, J. Oper. Res. Soc..

[100]  Enno Siemsen,et al.  Behavioral operations: The state of the field , 2013 .

[101]  Haiyan Song,et al.  Combining statistical and judgmental forecasts via a web-based tourism demand forecasting system , 2013 .

[102]  K. Wöber,et al.  Judgemental Forecasting Support Systems in Tourism , 2011 .

[103]  Marcus O'Connor,et al.  Time series characteristics and the widths of judgemental confidence intervals , 1992 .

[104]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[105]  M. O'Connor,et al.  Judgemental adjustment of initial forecasts: Its effectiveness and biases , 1995 .

[106]  Philip Hans Franses,et al.  Experts' Stated Behavior , 2009, Interfaces.

[107]  Robert D. Klassen,et al.  Forecasting practices of Canadian firms: Survey results and comparisons , 2001 .

[108]  Keith J. Burnham,et al.  Fuzzy decision support system for demand forecasting with a learning mechanism , 2006, Fuzzy Sets Syst..

[109]  Marcus O'Connor,et al.  Judgement or models: The importance of task differences , 1996 .

[110]  Robert J. Genetski,et al.  Long-Range Forecasting: From Crystal Ball to Computer , 1981 .

[111]  Li-Chih Wang,et al.  A GA-Based Sales Forecasting Model Incorporating Promotion Factors , 2010 .

[112]  R. Fildes,et al.  Adjusting supply chain forecasts for short-term temperature estimates: a case study in a Brewing company , 2013 .

[113]  Kai H. Lim,et al.  Improving judgmental forecasts with judgmental bootstrapping and task feedback support , 2005 .

[114]  Joa Sang Lim,et al.  Judgmental forecasting with time series and causal information , 1996 .

[115]  Gary P. Pisano,et al.  Toward a Theory of Behavioral Operations , 2008, Manuf. Serv. Oper. Manag..

[116]  Enno Siemsen,et al.  Effective judgmental forecasting in the context of fashion products , 2015 .

[117]  Laureen A. Maines.  An experimental examination of subjective forecast combination , 1996 .

[118]  Michael Lawrence,et al.  The effects of structural characteristics of explanations on use of a DSS , 2006, Decis. Support Syst..

[119]  S. Fawcett,et al.  Data Science, Predictive Analytics, and Big Data: A Revolution that Will Transform Supply Chain Design and Management , 2013 .

[120]  B. Ratchford,et al.  Practice Prize Article-CHAN4CAST: A Multichannel, Multiregion Sales Forecasting Model and Decision Support System for Consumer Packaged Goods , 2005 .

[121]  Harald J. van Heerde,et al.  How Promotions Work: Scan Pro-Based Evolutionary Model Building , 2002 .

[122]  Philip Hans Franses,et al.  Combining SKU-level sales forecasts from models and experts , 2009, Expert Syst. Appl..

[123]  Ananth Iyer,et al.  The Supply Chain Impact of Smart Customers in a Promotional Environment , 2002, Manuf. Serv. Oper. Manag..

[124]  Robert Fildes,et al.  Forecasting support systems: What we know, what we need to know , 2013 .

[125]  Larry P. Ritzman,et al.  Integrating judgmental and quantitative forecasts: methodologies for pooling marketing and operations information , 2004 .

[126]  Paul Goodwin,et al.  Adjusting Judgemental Extrapolations using Theil's Method and Discounted Weighted Regression , 1997 .

[127]  H. R. White,et al.  Sales Forecasting: Timesaving and Profit-Making Strategies That Work , 1984 .

[128]  Sunil Gupta,et al.  Managerial judgment and forecast combination: An experimental study , 1994 .

[129]  Bijan Fazlollahi,et al.  Adaptive decision support systems , 1997, Decis. Support Syst..

[130]  Fred Collopy,et al.  An Application of Rule-Based Forecasting to a Situation Lacking Domain Knowledge , 2000 .

[131]  Robert Fildes,et al.  Influence of user participation on DSS use and decision accuracy , 2002 .

[132]  Philip Hans Franses,et al.  Do Experts' SKU Forecasts Improve after Feedback? , 2011 .

[133]  Carl R. Anderson,et al.  Long-Range Forecasting: From Crystal Ball to Computer , 1979 .

[134]  Nigel Harvey,et al.  Combining forecasts: What information do judges need to outperform the simple average? , 1999 .

[135]  Craig R. Carter,et al.  Behavioral supply management: a taxonomy of judgment and decision‐making biases , 2007 .

[136]  P. Goodwin,et al.  Improving judgmental time series forecasting: A review of the guidance provided by research , 1993 .

[137]  Paul Goodwin,et al.  Statistical correction of judgmental point forecasts and decisions , 1996 .

[138]  Konstantinos Nikolopoulos,et al.  Supply chain forecasting: Theory, practice, their gap and the future , 2016, Eur. J. Oper. Res..

[139]  A. Adams.  Procedures for revising management judgments forecasts , 1986 .

[140]  Fotios Petropoulos,et al.  Relative performance of methods for forecasting special events , 2015 .

[141]  Philip Hans Franses,et al.  Do Experts’ SKU Forecasts Improve after Feedback?: Do Experts' SKU Forecasts Improve after Feedback? , 2014 .

[142]  Paul Goodwin,et al.  Antecedents and effects of trust in forecasting advice , 2013 .