Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions

Many applications in decision making under uncertainty and probabilistic risk assessment require the assessment of multiple, dependent uncertain quantities, so that in addition to marginal distributions, interdependence needs to be modelled in order to properly understand the overall risk. Nevertheless, relevant historical data on dependence information are often not available or simply too costly to obtain. In this case, the only sensible option is to elicit this uncertainty through the use of expert judgements. In expert judgement studies, a structured approach to eliciting variables of interest is desirable so that their assessment is methodologically robust. One of the key decisions during the elicitation process is the form in which the uncertainties are elicited. This choice is subject to various, potentially conflicting, desiderata related to e.g. modelling convenience, coherence between elicitation parameters and the model, combining judgements, and the assessment burden for the experts. While extensive and systematic guidance to address these considerations exists for single variable uncertainty elicitation, for higher dimensions very little such guidance is available. Therefore, this paper offers a systematic review of the current literature on eliciting dependence. The literature on the elicitation of dependence parameters such as correlations is presented alongside commonly used dependence models and experience from case studies. From this, guidance about the strategy for dependence assessment is given and gaps in the existing research are identified to determine future directions for structured methods to elicit dependence.

[1]  C. Genest,et al.  The Advent of Copulas in Finance , 2009 .

[2]  Roger M. Cooke,et al.  Uncertainty analysis comes to integrated assessment models for climate change…and conversely , 2013, Climatic Change.

[3]  J. René van Dorp,et al.  Statistical dependence through common risk factors: With applications in uncertainty analysis , 2005, European Journal of Operational Research.

[4]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[5]  Kerrie L. Mengersen,et al.  Elicitator: An expert elicitation tool for regression in ecology , 2010, Environ. Model. Softw..

[6]  D. V. Gokhale,et al.  Assessment of a Prior Distribution for the Correlation Coefficient in a Bivariate Normal Distribution , 1982 .

[7]  David Jenkinson,et al.  The Elicitation of Probabilities-A Review of the Statistical Literature , 2005 .

[8]  Jean-Luc Marichal,et al.  Tolerant or intolerant character of interacting criteria in aggregation by the Choquet integral , 2004, Eur. J. Oper. Res..

[9]  Christian List,et al.  Aggregating Causal Judgments , 2014, Philosophy of Science.

[10]  Gerd Gigerenzer,et al.  Communicating Statistical Information , 2000, Science.

[11]  Max Henrion,et al.  Some Practical Issues in Constructing Belief Networks , 1987, UAI.

[12]  Thierry Candresse,et al.  Scientific Opinion on the risk to plant health posed by Daktulosphaira vitifoliae (Fitch) in the EU territory, with the identification and evaluation of risk reduction options , 2014 .

[13]  C. Genest,et al.  Everything You Always Wanted to Know about Copula Modeling but Were Afraid to Ask , 2007 .

[14]  K. McConway Marginalization and Linear Opinion Pools , 1981 .

[15]  F. Lad Operational Subjective Statistical Methods: A Mathematical, Philosophical, and Historical Introduction , 1996 .

[16]  Luca Regis,et al.  A Bayesian copula model for stochastic claims reserving , 2011 .

[17]  Reid Hastie,et al.  Causal Thinking in Judgments , 2015 .

[18]  Prakash P. Shenoy,et al.  A causal mapping approach to constructing Bayesian networks , 2004, Decis. Support Syst..

[19]  Dorota Kurowicka,et al.  Eliciting conditional and unconditional rank correlations from conditional probabilities , 2008, Reliab. Eng. Syst. Saf..

[20]  Erim Kardes,et al.  Calibration, sharpness and the weighting of experts in a linear opinion pool , 2015, Ann. Oper. Res..

[21]  Detlof von Winterfeldt,et al.  Anniversary Article: Decision Analysis in Management Science , 2004, Manag. Sci..

[22]  Salvatore Greco,et al.  Rough sets theory for multicriteria decision analysis , 2001, Eur. J. Oper. Res..

[23]  Adnan Darwiche,et al.  Modeling and Reasoning with Bayesian Networks , 2009 .

[24]  P. Garthwaite,et al.  An elicitation method for multivariate normal distributions , 1998 .

[25]  Oswaldo Morales-Nápoles,et al.  Non-parametric Bayesian networks: Improving theory and reviewing applications , 2015, Reliab. Eng. Syst. Saf..

[26]  Anthony O'Hagan,et al.  Elicitation of multivariate prior distributions: A nonparametric Bayesian approach , 2010 .

[27]  Fadlalla G. Elfadaly,et al.  Prior distribution elicitation for generalized linear and piecewise-linear models , 2013 .

[28]  Ali E. Abbas,et al.  Multiattribute Utility Copulas , 2009, Oper. Res..

[29]  George T. Duncan,et al.  Some properties of the dirichlet-multinomial distribution and its use in prior elicitation , 1987 .

[30]  Nils Blomqvist,et al.  On a Measure of Dependence Between two Random Variables , 1950 .

[31]  Oswaldo Morales-Nápoles,et al.  A continuous Bayesian network for earth dams' risk assessment: methodology and quantification , 2014 .

[32]  Robert L. Winkler,et al.  Multiple Experts vs. Multiple Methods: Combining Correlation Assessments , 2004, Decis. Anal..

[33]  Holger Fink,et al.  Bayesian Risk Aggregation: Correlation Uncertainty and Expert Judgement , 2010 .

[34]  Ali E. Abbas,et al.  Assessing Joint Distributions with Isoprobability Contours , 2010, Manag. Sci..

[35]  Andy Hart,et al.  A Bayes Linear Approach to Weight-of-Evidence Risk Assessment for Skin Allergy , 2013 .

[36]  Ruth Beyth-Marom,et al.  Perception of correlation reexamined , 1982 .

[37]  J. Lee,et al.  Post Processing Techniques , 2000 .

[38]  Fumika Ouchi,et al.  A literature review on the use of expert opinion in probabilistic risk analysis , 2004 .

[39]  Paul H. Garthwaite,et al.  An elicitation method for multiple linear regression models , 1991 .

[40]  Kalyanmoy Deb,et al.  Multiple Criteria Decision Making, Multiattribute Utility Theory: Recent Accomplishments and What Lies Ahead , 2008, Manag. Sci..

[41]  Thomas A. Louis,et al.  Graphical Elicitation of a Prior Distribution for a Clinical Trial , 1993 .

[42]  Marek J. Druzdzel,et al.  Elicitation of Probabilities for Belief Networks: Combining Qualitative and Quantitative Information , 1995, UAI.

[43]  Tim Bedford,et al.  Probabilistic Inversion of Expert Judgments in the Quantification of Model Uncertainty , 2005, Manag. Sci..

[44]  Lesley Walls,et al.  Supporting Reliability Decisions During Defense Procurement Using a Bayes Linear Methodology , 2011, IEEE Transactions on Engineering Management.

[45]  Luca Podofillini,et al.  Bayesian belief networks for human reliability analysis: A review of applications and gaps , 2015, Reliab. Eng. Syst. Saf..

[46]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[47]  F. Dietrich,et al.  Probabilistic Opinion Pooling , 2016 .

[48]  David R. Shanks,et al.  Judging Covariation and Causation , 2008 .

[49]  Christopher J. Mitchell,et al.  The propositional nature of human associative learning , 2009, Behavioral and Brain Sciences.

[50]  Albert Y. Kim,et al.  Hypothesis Testing , 2019, Encyclopedic Dictionary of Archaeology.

[51]  Lesley Walls,et al.  Evaluation of elicitation methods to quantify Bayes linear models , 2010 .

[52]  Roger M. Cooke,et al.  The anatomy of the squizzel: The role of operational definitions in representing uncertainty , 2004, Reliab. Eng. Syst. Saf..

[53]  大西 仁,et al.  Pearl, J. (1988, second printing 1991). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan-Kaufmann. , 1994 .

[54]  Herbert Moskowitz,et al.  Improving the Consistency of Conditional Probability Assessments for Forecasting and Decision Making , 1983 .

[55]  S. Weisberg Applied Linear Regression , 1981 .

[56]  L. C. van der Gaag,et al.  Building probabilistic networks: Where do the numbers come from? - a guide to the literature , 2000 .

[57]  Robert L. Winkler,et al.  Sensitivity to Distance and Baseline Distributions in Forecast Evaluation , 2009, Manag. Sci..

[58]  Stephen E. Fienberg,et al.  The Comparison and Evaluation of Forecasters. , 1983 .

[59]  L. J. Chapman,et al.  Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. , 1969, Journal of abnormal psychology.

[60]  Kerrie Mengersen,et al.  Elicitation by design in ecology: using expert opinion to inform priors for Bayesian statistical models. , 2009, Ecology.

[61]  Silja Renooij,et al.  Probability elicitation for belief networks: issues to consider , 2001, The Knowledge Engineering Review.

[62]  Wayne S. Smith,et al.  Interactive Elicitation of Opinion for a Normal Linear Model , 1980 .

[63]  M. Farrow Practical building of subjective covariance structures for large complicated systems , 2003 .

[64]  Ana Isabel Barros,et al.  Relieving the elicitation burden of Bayesian Belief Networks , 2008, BMA.

[65]  Roger Cooke,et al.  Validating Expert Judgment with the Classical Model , 2014 .

[66]  R. Clemen,et al.  Correlations and Copulas for Decision and Risk Analysis , 1999 .

[67]  J. Kadane,et al.  Experiences in elicitation , 1998 .

[68]  Mkael Symmonds,et al.  Hedging Your Bets by Learning Reward Correlations in the Human Brain , 2011, Neuron.

[69]  Roger M. Cooke,et al.  Expert judgment based multi-criteria decision model to address uncertainties in risk assessment of nanotechnology-enabled food products , 2011 .

[70]  Norman E. Fenton,et al.  Using Ranked Nodes to Model Qualitative Judgments in Bayesian Networks , 2007, IEEE Transactions on Knowledge and Data Engineering.

[71]  O. Morales Nápoles,et al.  Bayesian Belief Nets and Vines in Aviation Safety and other Applications , 2010 .

[72]  Roger M. Cooke,et al.  Post-processing techniques for the joint CEC/USNRC uncertainty analysis of accident consequence codes , 1997 .

[73]  D. V. Lindley,et al.  Bayesian estimation of the dispersion matrix of a multivariate normal distribution , 1985 .

[74]  Herbert Moskowitz,et al.  Decision and risk analysis , 1987 .

[75]  Tim Bedford,et al.  Approximate Uncertainty Modeling in Risk Analysis with Vine Copulas , 2015, Risk analysis : an official publication of the Society for Risk Analysis.

[76]  P. H. Garthwaite,et al.  Prior distribution assessment for a multivariate normal distribution: An experimental study , 2001 .

[77]  David F. Percy,et al.  Bayesian enhanced strategic decision making for reliability , 2002, Eur. J. Oper. Res..

[78]  Paul H. Garthwaite,et al.  Non‐conjugate prior distribution assessment for multivariate normal sampling , 2001 .

[79]  Detlof von Winterfeldt,et al.  Structuring decision problems: A case study and reflections for practitioners , 2009, Eur. J. Oper. Res..

[80]  George Wright,et al.  Assessing the quality of expert judgment: Issues and analysis , 1994, Decis. Support Syst..

[81]  Fadlalla G. Elfadaly,et al.  Eliciting Dirichlet and Connor–Mosimann prior distributions for multinomial models , 2013 .

[82]  W. Kruskal Ordinal Measures of Association , 1958 .

[83]  Roger M. Cooke,et al.  Modeling and validating stakeholder preferences with probabilistic inversion , 2011 .

[84]  R. Cooke Parameter fitting for uncertain models: modelling uncertainty in small models , 1994 .

[85]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[86]  Tim Bedford,et al.  The effect of model uncertainty on maintenance optimization , 2002, IEEE Trans. Reliab..

[87]  Robert L. Winkler,et al.  Scoring Rules, Generalized Entropy, and Utility Maximization , 2008, Oper. Res..

[88]  Philip M. Lurie,et al.  An Approximate Method for Sampling Correlated Random Variables From Partially-Specified Distributions , 1998 .

[89]  Salvatore Greco,et al.  Multi-criteria classification - A new scheme for application of dominance-based decision rules , 2007, Eur. J. Oper. Res..

[90]  Kathrin Klamroth,et al.  An MCDM approach to portfolio optimization , 2004, Eur. J. Oper. Res..

[91]  David A. Schkade,et al.  WHERE DO THE NUMBERS COME FROM , 1993 .

[92]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[93]  R. Cooke Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[94]  T. Bedford,et al.  Probabilistic Risk Analysis: Foundations and Methods , 2001 .

[95]  Michail Papathomas,et al.  Updating beliefs for binary variables , 2005 .

[96]  Oswaldo Morales-Nápoles,et al.  A continuous Bayesian network for earth dams’ risk assessment: an application , 2014 .

[97]  David Wooff,et al.  Bayes Linear Statistics: Theory and Methods , 2007 .

[98]  Collin Carbno,et al.  Uncertainty Analysis With High Dimensional Dependence Modelling , 2007, Technometrics.

[99]  Francisco Javier Díez,et al.  Parameter adjustment in Bayes networks. The generalized noisy OR-gate , 1993, UAI.

[100]  A. D. Swain,et al.  Handbook of human-reliability analysis with emphasis on nuclear power plant applications. Final report , 1983 .

[101]  Davide Canestraro,et al.  Estimating Copulas for Insurance from Scarce Observations, Expert Opinion and Prior Information: A Bayesian Approach , 2012, ASTIN Bulletin.

[102]  Martin Odening,et al.  Can expert knowledge compensate for data scarcity in crop insurance pricing , 2016 .

[103]  Oswaldo Morales-Nápoles,et al.  Experimental results about the assessments of conditional rank correlations by experts: Example with air pollution estimates , 2013 .

[104]  L. Allan A note on measurement of contingency between two binary variables in judgment tasks , 1980 .

[105]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[106]  D. Keefer,et al.  Three-Point Approximations for Continuous Random Variables , 1983 .

[107]  Roger M. Cooke,et al.  Dealing with Dependencies in Uncertainty Analysis , 1996 .

[108]  Adrianus M. H. Meeuwissen,et al.  Minimally informative distributions with given rank correlation for use in uncertainty analysis , 1997 .

[109]  M. Grabisch The application of fuzzy integrals in multicriteria decision making , 1996 .

[110]  Roger M. Cooke,et al.  Climate Change Uncertainty Quantification: Lessons Learned from the Joint EU-USNRC Project on Uncertainty Analysis of Probabilistic Accident Consequence Codes , 2010 .

[111]  Joachim Meyer,et al.  Correlation estimates as perceptual judgments. , 1997 .

[112]  R. Nisbett,et al.  The psychometrics of everyday life , 1986, Cognitive Psychology.

[113]  P. Embrechts,et al.  Dependence modeling with copulas , 2007 .

[114]  J. Smedslund THE CONCEPT OF CORRELATION IN ADULTS , 1963 .

[115]  J. Schreiber Foundations Of Statistics , 2016 .

[116]  Silja Renooij,et al.  How to Elicit Many Probabilities , 1999, UAI.

[117]  Salvatore Greco,et al.  Assessing non-additive utility for multicriteria decision aid , 2004, Eur. J. Oper. Res..

[118]  Salvatore Greco,et al.  Axiomatic characterization of a general utility function and its particular cases in terms of conjoint measurement and rough-set decision rules , 2004, Eur. J. Oper. Res..

[119]  José Rui Figueira,et al.  Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method , 2009, Eur. J. Oper. Res..

[120]  Kerrie Mengersen,et al.  Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush‐tailed rock‐wallaby Petrogale penicillata , 2009 .

[121]  Salvatore Greco,et al.  Robust ordinal regression for value functions handling interacting criteria , 2014, Eur. J. Oper. Res..

[122]  Terence Reilly,et al.  Sensitivity Analysis for Dependent Variables , 2000, Decis. Sci..

[123]  Simon French,et al.  Aggregating expert judgement , 2011 .

[124]  M. Sklar Fonctions de repartition a n dimensions et leurs marges , 1959 .

[125]  Roger M. Cooke,et al.  TU Delft expert judgment data base , 2008, Reliab. Eng. Syst. Saf..

[126]  Robert L. Winkler,et al.  Assessing Dependence: Some Experimental Results , 2000 .

[127]  Maria Hänninen,et al.  Bayesian network model of maritime safety management , 2014, Expert Syst. Appl..

[128]  B.C.P. Kraan,et al.  Probabilistic inversion in uncertainty analysis: And related topics , 2002 .

[129]  Tim Bedford,et al.  Interactive expert assignment of minimally-informative copulae , 2002 .

[130]  Thomas P. Ryan,et al.  Modern Regression Methods , 1996 .

[131]  Paul L. Fackler,et al.  Modeling Interdependence: An Approach to Simulation and Elicitation , 1991 .

[132]  N. Balakrishnan,et al.  A Primer on Statistical Distributions , 2003 .

[133]  Felix Salmon The formula that killed Wall Street , 2012 .

[134]  Samuel Kotz,et al.  Generalized Diagonal Band Copulas with Two-Sided Generating Densities , 2010, Decis. Anal..

[135]  Didier Dubois,et al.  Decision-theoretic foundations of qualitative possibility theory , 2001, Eur. J. Oper. Res..

[136]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[137]  David F. Percy Subjective priors for maintenance models , 2004 .

[138]  C. Sempi,et al.  Principles of Copula Theory , 2015 .

[139]  Anthony O'Hagan,et al.  Subjective Modelling and Bayes Linear Estimation in the UK Water Industry , 1992 .

[140]  Anders Winman,et al.  Rational Assessments of Covariation and Causality , 2000 .

[141]  Marek J. Druzdzel,et al.  An Empirical Study of Probability Elicitation Under Noisy-OR Assumption , 2004, FLAIRS.

[142]  E. S. Pearson,et al.  Approximate means and standard deviations based on distances between percentage points of frequency curves , 1965 .

[143]  Paul H. Garthwaite,et al.  Quantifying expert opinion for modelling fauna habitat distributions , 2006, Comput. Stat..

[144]  Anthony O'Hagan,et al.  Eliciting expert judgements about a set of proportions , 2014 .

[145]  T. Bedford,et al.  Applying Bayes linear methods to support reliability procurement decisions , 2008, 2008 Annual Reliability and Maintainability Symposium.

[146]  F. D. L. Torre,et al.  Designing a Metric for the Difference between Gaussian Densities , 2010 .

[147]  R. Christensen,et al.  A New Perspective on Priors for Generalized Linear Models , 1996 .