Advances in Decision Analysis: Aggregating Probability Distributions

ABSTRACT . This chapter is concerned with the aggregation of probability distributions in decision and risk analysis. Experts often provide valuable information regarding important uncertainties in decision and risk analyses because of the limited availability of hard data to use in those analyses. Multiple experts are often consulted in order to obtain as much information as possible, leading to the problem of how to combine or aggregate their information. Information may also be obtained from other sources such as forecasting techniques or scientific models. Because uncertainties are typically represented in terms of probability distributions, we consider expert and other information in terms of probability distributions. We discuss a variety of models that lead to specific combination methods. The output of these methods is a combined probability distribution , which can be viewed as representing a summary of the current state of information regarding the uncertainty of interest. After presenting the models and methods, we discuss empirical evidence on the performance of the methods. In the conclusion, we highlight important conceptual and practical issues to be considered when designing a combination process for use in practice. Introduction Expert judgments can provide useful information for forecasting, making decisions, and assessing risks. Such judgments have been used informally for many years. In recent years, the use of formal methods to combine expert judgments has become increasingly commonplace. Cooke (1991) reviews many of the developments over the years as attempts have been made to use expert judgments in various settings.

[1]  B. D. Finetti La prévision : ses lois logiques, ses sources subjectives , 1937 .

[2]  L. A. Goodman,et al.  Social Choice and Individual Values , 1951 .

[3]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[4]  M. Stone The Opinion Pool , 1961 .

[5]  Robert L. Winkler,et al.  The Consensus of Subjective Probability Distributions , 1968 .

[6]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[7]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[8]  Barbara C. Goodman,et al.  Action selection and likelihood ratio estimation by individuals and groups , 1972 .

[9]  Carl-Axel S. Staël von Holstein,et al.  Probabilistic forecasting: An experiment related to the stock market , 1972 .

[10]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[11]  C. Granger,et al.  Experience with Forecasting Univariate Time Series and the Combination of Forecasts , 1974 .

[12]  J. Armstrong The Use of the Decomposition Principle in Making Judgments , 1975 .

[13]  D. W. Bunn,et al.  A Bayesian Approach to the Linear Combination of Forecasts , 1975 .

[14]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[15]  Peter A. Morris,et al.  Combining Expert Judgments: A Bayesian Approach , 1977 .

[16]  D. A. Seaver Assessing Probability with Multiple Individuals: Group Interaction Versus Mathematical Aggregation. , 1978 .

[17]  M. Bacharach Normal Bayesian Dialogues , 1979 .

[18]  John Rohrbaugh,et al.  Improving the quality of group judgment: Social judgment analysis and the Delphi technique. , 1979 .

[19]  Simon French,et al.  Consensus of opinion , 1981 .

[20]  R. L. Winkler Combining Probability Distributions from Dependent Information Sources , 1981 .

[21]  D. V. Gokhale,et al.  Assessment of a Prior Distribution for the Correlation Coefficient in a Bivariate Normal Distribution , 1982 .

[22]  G. W. Hill Group versus individual performance: are n + 1 heads better than one?" psychological bulletin , 1982 .

[23]  R. L. Winkler,et al.  The Combination of Forecasts , 1983 .

[24]  Peter A. Morris,et al.  An Axiomatic Approach to Expert Resolution , 1983 .

[25]  Dennis V. Lindley,et al.  Reconciliation of Probability Distributions , 1983, Oper. Res..

[26]  C. Genest Pooling operators with the marginalization property , 1984 .

[27]  Robert L. Winkler,et al.  Limits for the Precision and Value of Information from Dependent Sources , 1985, Oper. Res..

[28]  Christian Genest,et al.  Modeling Expert Judgments for Bayesian Updating , 1985 .

[29]  R. H. Edmundson,et al.  The accuracy of combining judgemental and statistical forecasts , 1986 .

[30]  Robert L. Winkler,et al.  Expert resolution , 1986 .

[31]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[32]  Bernard Grofman,et al.  Information pooling and group decision making : proceedings of the Second University of California, Irvine, Conference on Political Economy , 1986 .

[33]  M. Degroot,et al.  Bayesian Statistics 2. , 1987 .

[34]  Reinhard Viertl,et al.  Probability and Bayesian Statistics , 1987 .

[35]  Robert L. Winkler,et al.  Calibrating and Combining Precipitation Probability Forecasts , 1987 .

[36]  M. W. Merkhofer,et al.  Quantifying judgmental uncertainty: Methodology, experiences, and insights , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[37]  Peter Ayton,et al.  The consistency, coherence and calibration of holistic, decomposed and recomposed judgemental probability forecasts , 1988 .

[38]  Paul Slovic,et al.  Structuring Knowledge Retrieval: An Analysis of Decomposed Quantitative Judgments , 1988 .

[39]  Ali Mosleh,et al.  A critique of current practice for the use of expert opinions in probabilistic risk assessment , 1988 .

[40]  R. Dawes,et al.  Heuristics and Biases: Clinical versus Actuarial Judgment , 2002 .

[41]  Ralph L. Keeney,et al.  On the uses of expert judgment on complex technical problems , 1989 .

[42]  Janet A. Sniezek,et al.  An examination of group process in judgmental forecasting , 1989 .

[43]  Edna M. White,et al.  Subjective versus objective combining of forecasts: An experiment , 1989 .

[44]  Ronald L. Iman,et al.  Expert opinion in risk analysis: the NUREG-1150 methodology , 1989 .

[45]  R. Clemen Combining forecasts: A review and annotated bibliography , 1989 .

[46]  Christian Genest,et al.  Allocating the weights in the linear opinion pool , 1990 .

[47]  G. Apostolakis,et al.  Uncertainties in system analysis: Probabilistic versus nonprobabilistic theories , 1990 .

[48]  R. L. Winkler,et al.  Unanimity and compromise among probability forecasters , 1990 .

[49]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[50]  David C. Schmittlein,et al.  Combining Forecasts: Operational Adjustments to Theoretically Optimal Rules , 1990 .

[51]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[52]  D. Bunn,et al.  Interaction of judgemental and statistical forecasting methods: issues & , 1991 .

[53]  R. Cooke Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[54]  Samuel Kotz,et al.  Advances in probability distributions with given marginals : beyond the copulas , 1991 .

[55]  H Otway,et al.  Expert judgment in risk analysis and management: process, context, and pitfalls. , 1992, Risk analysis : an official publication of the Society for Risk Analysis.

[56]  M. West,et al.  Modelling Probabilistic Agent Opinion , 1992 .

[57]  Robert L. Winkler,et al.  Sensitivity of Weights in Combining Forecasts , 1992, Oper. Res..

[58]  Stephen C. Hora,et al.  Acquisition of Expert Judgment: Examples from Risk Assessment , 1992 .

[59]  M. West,et al.  Modelling Agent Forecast Distributions , 1992 .

[60]  J. H. Davis,et al.  Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples 1950-1990 , 1992 .

[61]  Nancy G. Dodd,et al.  The use of decomposition in probability assessments of continuous variables , 1993 .

[62]  George Apostolakis,et al.  Some approximations useful to the use of dependent information sources , 1993 .

[63]  Robert L. Winkler,et al.  Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit , 1993 .

[64]  Robert L. Winkler,et al.  Aggregating Point Estimates: A Flexible Modeling Approach , 1993 .

[65]  Gordon P. Wright,et al.  Testing for unreliable estimators and insignificant forecasts in combined forecasts , 1994 .

[66]  A. I. Shlyakhter,et al.  An improved framework for uncertainty analysis: Accounting for unsuspected errors , 1994 .

[67]  A. I. Shlyakhter,et al.  Quantifying the credibility of energy projections from trends in past data: the US energy sector , 1994 .

[68]  P. Reagan-Cirincione,et al.  Improving the Accuracy of Group Judgment: A Process Intervention Combining Group Facilitation, Social Judgment Analysis, and Information Technology , 1994 .

[69]  A. Gelfand,et al.  Modeling Expert Opinion Arising as a Partial Probabilistic Specification , 1995 .

[70]  Robert L. Winkler,et al.  An Assessment of the Risk of Chronic Lung Injury Attributable to Long-Term Ozone Exposure , 1995, Oper. Res..

[71]  M. G. Morgan,et al.  Subjective judgments by climate experts. , 1995 .

[72]  Robert T. Clemen,et al.  Copula Models for Aggregating Expert Opinions , 1996, Oper. Res..

[73]  Robert L. Winkler,et al.  Aggregating Forecasts: An Empirical Evaluation of Some Bayesian Methods , 1996 .

[74]  R. Hastie,et al.  Proper analysis of the accuracy of group judgments , 1997 .

[75]  Giovanni Parmigiani,et al.  Combining Expert Judgment by Hierarchical Modeling: An Application to Physician Staffing , 1998 .

[76]  R. Clemen,et al.  Correlations and Copulas for Decision and Risk Analysis , 1999 .

[77]  James K. Hammitt,et al.  The Expected Value of Information and the Probability of Surprise , 1999 .

[78]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[79]  Simon French,et al.  Statistical Decision Theory , 1980 .

[80]  Robert L. Winkler,et al.  Assessing Dependence: Some Experimental Results , 2000 .

[81]  J. Scott Armstrong,et al.  Principles of forecasting : a handbook for researchers and practitioners , 2001 .

[82]  Christopher J. Lacke,et al.  Analysis of Colorectal Cancer Screening Regimens , 2001, Health Care Management Science.

[83]  Richard S. John,et al.  Assessment of Cost Uncertainties for Large Technology Projects: A Methodology and an Application , 2002, Interfaces.

[84]  Stephen C. Hora,et al.  Probability Judgments for Continuous Quantities: Linear Combinations and Calibration , 2004, Manag. Sci..

[85]  Robert L. Winkler,et al.  Multiple Experts vs. Multiple Methods: Combining Correlation Assessments , 2004, Decis. Anal..

[86]  D. Read Judgment and Choice , 2005 .

[87]  A. O'Hagan,et al.  Statistical Methods for Eliciting Probability Distributions , 2005 .

[88]  Jason R. W. Merrick,et al.  Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons , 2005, Decis. Anal..

[89]  Richard P. Larrick,et al.  Intuitions About Combining Opinions: Misappreciation of the Averaging Principle , 2006, Manag. Sci..