Reaction to ‘An approach to perform expert elicitation for engineering design risk analysis: methodology and experimental results’

type="main" xml:id="rssa12028-abs-0001"> Expert elicitation is increasingly applied to different research areas. Multiple approaches have been implemented, but the development of methods to quantify experts' biases and calibration represents a challenge. As a result, the integration of multiple and often conflicting opinions can be demanding, owing to the complexity of properly weighting experts' contributions. We propose an approach to address this problem when probability densities for seed calibration variables are not available. The methodology generates an expert score that is employed to aggregate multiple-expert assessments. The approach has been experimentally applied to engineering design risk analysis. Results indicate that the approach improves the quality of the estimations. The weighted aggregations of experts' estimates based on the experts' scores achieve better results than the corresponding aggregations based on experts' opinions equally weighted.

[1]  John Quigley,et al.  Building prior distributions to support Bayesian reliability growth modelling using expert judgement , 2001, Reliab. Eng. Syst. Saf..

[2]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[3]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[4]  Roger M. Cooke,et al.  TU Delft expert judgment data base , 2008, Reliab. Eng. Syst. Saf..

[5]  M. W. Merkhofer,et al.  Quantifying judgmental uncertainty: Methodology, experiences, and insights , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[6]  Stephen C. Hora,et al.  Probability Judgments for Continuous Quantities: Linear Combinations and Calibration , 2004, Manag. Sci..

[7]  Nicholas Frank Pidgeon,et al.  The conjunction fallacy: The case for the existence of competing heuristic strategies , 1997 .

[8]  William F. Wright,et al.  Effects of situation familiarity and financial incentives on use of the anchoring and adjustment heuristic for probability assessment , 1989 .

[9]  Robert A. Jacobs,et al.  Methods For Combining Experts' Probability Assessments , 1995, Neural Computation.

[10]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[11]  A. Tversky,et al.  On the Reconciliation of Probability Assessments , 1979 .

[12]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[13]  Roger M. Cooke,et al.  On the performance of social network and likelihood-based expert weighting schemes , 2008, Reliab. Eng. Syst. Saf..

[14]  C. Murray,et al.  Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research , 2003, American Political Science Review.

[15]  Alessandra Babuscia,et al.  Statistical Risk Estimation for Communication System Design , 2013, IEEE Systems Journal.

[16]  Anthony O'Hagan,et al.  Eliciting expert beliefs in substantial practical applications , 1998 .

[17]  J. Koehler.  The base rate fallacy reconsidered: Descriptive, normative, and methodological challenges , 1996, Behavioral and Brain Sciences.

[18]  Dennis V. Lindley,et al.  Reconciliation of Probability Distributions , 1983, Oper. Res..

[19]  M. J. Kallen,et al.  EXPERT AGGREGATION WITH DEPENDENCE , 2002 .

[20]  Joseph H. Yuen,et al.  Deep Space Telecommunications Systems Engineering , 2013 .

[21]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[22]  Harold I. Jacobson.  THE MAXIMUM VARIANCE OF RESTRICTED UNIMODAL DISTRIBUTIONS , 1969 .

[23]  John Quigley,et al.  Expert judgement combination using moment methods , 2008, Reliab. Eng. Syst. Saf..

[24]  A. Tversky,et al.  Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment , 1983 .

[25]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[26]  Franco Flandoli,et al.  Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique , 2011, Reliab. Eng. Syst. Saf..