Use of expert knowledge to anticipate the future: Issues, analysis and directions

Unless an anticipation problem is routine and short-term, and objective data are plentiful, expert judgment will be needed. Risk assessment is analogous to anticipating the future, in that models need to be developed and applied to data. Since objective data are often scanty, expert knowledge elicitation (EKE) techniques have been developed for risk assessment that allow models to be developed and parametrized using expert judgment with minimal cognitive and social biases. Here, we conceptualize how EKE can be developed and applied to support anticipation of the future. Accordingly, we begin by defining EKE as a complete process, which involves considering experts as a source of data, and comprises various methods for ensuring the quality of this data, including selecting the best experts, training experts in the normative aspects of anticipation, and combining judgments from several experts, as well as eliciting unbiased estimates and constructs from experts. We detail various aspects of the papers that constitute this special issue and analyse them in terms of the stages of the EKE future-anticipation process that they address. We also identify the remaining gaps in our knowledge. Our conceptualization of EKE with the aim of supporting anticipation of the future is compared and contrasted with the extant research on judgmental forecasting.

[1]  Philip Hans Franses,et al.  One model and various experts: Evaluating Dutch macroeconomic forecasts , 2011 .

[2]  Fotios Petropoulos,et al.  Using a rolling training approach to improve judgmental extrapolations elicited from forecasters with technical knowledge , 2017 .

[3]  Shashi Shekhar,et al.  Cognitive Mapping , 2008, Encyclopedia of GIS.

[4]  Marie-Line Germain,et al.  A preliminary exploration on the measurement of expertise: An initial development of a psychometric scale , 2012 .

[5]  James Surowiecki.  The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations Doubleday Books. , 2004 .

[6]  R. M. Oliver,et al.  Influence diagrams, belief nets and decision analysis , 1992 .

[7]  Torsten Wulf,et al.  Quantifiying blind spots and weak signals in executive judgment: A structured integration of expert judgment into the scenario development process , 2017 .

[8]  Henrik Olsson,et al.  Adaptive changes between cue abstraction and exemplar memory in a multiple-cue judgment task with continuous cues , 2007, Psychonomic bulletin & review.

[9]  George S. Day,et al.  How to Make Sense of Weak Signals , 2009 .

[10]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[11]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[12]  R. Cooke.  The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight? , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[13]  Roger Cooke,et al.  Validating Expert Judgment with the Classical Model , 2014 .

[14]  David V. Budescu,et al.  A review of human linguistic probability processing: General principles and empirical evidence , 1995, The Knowledge Engineering Review.

[15]  Robert J. Genetski,et al.  Long-Range Forecasting: From Crystal Ball to Computer , 1981 .

[16]  Gene Rowe,et al.  Final report on ‘the identification of food safety priorities using the Delphi technique’ , 2016 .

[17]  Kesten C. Green,et al.  Simple Versus Complex Forecasting: The Evidence , 2015 .

[18]  George Wright,et al.  Improving the Delphi process: Lessons from social psychological research , 2011 .

[19]  George S. Day,et al.  Integrating organizational networks, weak signals, strategic radars and scenario planning , 2013 .

[20]  Philip E. Tetlock,et al.  Superforecasting: The Art and Science of Prediction , 2015 .

[21]  Mark A. Burgman,et al.  Expert Status and Performance , 2011, PloS one.

[22]  Arthur E. Hoerl,et al.  Reliability of Subjective Evaluations in a High Incentive Situation , 1974 .

[23]  G. Stasser,et al.  Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion , 1985 .

[24]  Roger M. Cooke,et al.  On the performance of social network and likelihood-based expert weighting schemes , 2008, Reliab. Eng. Syst. Saf..

[25]  James Derbyshire,et al.  Augmenting the Intuitive Logics scenario planning method for a more comprehensive analysis of causation , 2017 .

[26]  Mandeep K. Dhami,et al.  Interpersonal comparison of subjective probabilities: Toward translating linguistic probabilities , 2005, Memory & cognition.

[27]  George Wright,et al.  Assessing the quality of expert judgment: Issues and analysis , 1994, Decis. Support Syst..

[28]  D. Broadbent,et al.  On the Relationship between Task Performance and Associated Verbalizable Knowledge , 1984 .

[29]  David R. Shanks.  Representation of categories and concepts in memory. , 1997 .

[30]  Magne Jørgensen,et al.  When 90% confidence intervals are 50% certain: on the credibility of credible intervals , 2005 .

[31]  Jeremy E. Oakley,et al.  Uncertain Judgements: Eliciting Experts' Probabilities , 2006 .

[32]  Erik Lebret,et al.  The use of expert elicitation in environmental health impact assessment: a seven step procedure , 2010, Environmental health : a global access science source.

[33]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[34]  Theodore J. Gordon,et al.  RT Delphi: An efficient, “round-less” almost real time Delphi method ☆ , 2006 .

[35]  Mark A. Filippell.  Identifying and Cultivating the Right Buyers , 2012 .

[36]  Shahram Sarkani,et al.  The effect of the number of seed variables on the performance of Cooke′s classical model , 2014, Reliab. Eng. Syst. Saf..

[37]  R. Fildes,et al.  Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning , 2009 .

[38]  Gene Rowe,et al.  There is data, and then there is data: only experimental evidence will determine the utility of differential weighting of expert judgment. , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[39]  P. Goodwin,et al.  Judgmental forecasting: A review of progress over the last 25 years , 2006 .

[40]  Mark A. Burgman,et al.  I nvestigate D iscuss E stimate A ggregate for structured expert judgement , 2017 .

[41]  John Paul Gosling,et al.  SHELF: The Sheffield Elicitation Framework , 2018 .

[42]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[43]  Nigel Harvey,et al.  Heuristics and biases in judgmental forecasting , 1998 .

[44]  J. Charles Kerkering,et al.  Eliciting and Analyzing Expert Judgment, A Practical Guide , 2002, Technometrics.

[45]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[46]  Jack T. Dennerlein,et al.  Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting , 2017 .

[47]  M. G. Morgan.  Use (and abuse) of expert elicitation in support of decision making for public policy , 2014, Proceedings of the National Academy of Sciences.

[48]  George R. Terre.  Influence Diagrams, Belief Nets, and Decision Analysis , 2012 .

[49]  Robert J. Budnitz,et al.  Recommendations for probabilistic seismic hazard analysis: Guidance on uncertainty and use of experts , 1997 .

[50]  R. Prinn,et al.  Uncertainty and Climate Change Assessments , 2001, Science.

[51]  Kevin J. Wilson,et al.  An investigation of dependence in expert judgement studies with multiple experts , 2017 .

[52]  Vicki M. Bier,et al.  A study of expert overconfidence , 2008, Reliab. Eng. Syst. Saf..

[53]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[54]  Paul Goodwin,et al.  Evaluating expert advice in forecasting: Users’ reactions to presumed vs. experienced credibility , 2017 .

[55]  Gideon Keren,et al.  Facing uncertainty in the game of bridge: A calibration study , 1987 .

[56]  Roger M Cooke,et al.  Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake Erie , 2014, Integrated environmental assessment and management.

[57]  A. Tversky,et al.  Choices, Values, and Frames , 2000 .

[58]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[59]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[60]  R. Cooke,et al.  Procedures Guide for Structural Expert Judgement in Accident Consequence Modelling , 2000 .

[61]  Thomas Langer,et al.  True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration , 2012 .

[62]  J. R. Larson,et al.  Diagnosing groups: the pooling, management, and impact of shared and unshared case information in team-based medical decision making. , 1998, Journal of personality and social psychology.

[63]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[64]  W. Aspinall.  A route to more tractable expert advice , 2010, Nature.

[65]  Thomas Langer,et al.  True Overconfidence in Interval Estimates: Evidence Based on a New Measure of Miscalibration: True Overconfidence in Interval Estimates , 2013 .

[66]  George Bernard Shaw,et al.  LONG-RANGE FORECASTING From Crystal Ball to Computer , 2010 .

[67]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[68]  Murray Turoff,et al.  The Delphi Method: Techniques and Applications , 1976 .

[69]  A. H. Murphy,et al.  A comparative evaluation of objective and subjective weather forecasts in the united states , 1984 .

[70]  Philip Tetlock,et al.  Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions , 2015, Perspectives on psychological science : a journal of the Association for Psychological Science.

[71]  Sydney E. Scott,et al.  Psychological Strategies for Winning a Geopolitical Forecasting Tournament , 2014, Psychological science.