Data-informed calibration and aggregation of expert opinion in a Bayesian framework

Title of Dissertation: DATA-INFORMED CALIBRATION AND AGGREGATION OF EXPERT JUDGMENT IN A BAYESIAN FRAMEWORK Calvin Homayoon Shirazi, Doctor of Philosophy, 2009 Dissertation Directed by: Professor Ali Mosleh, Rel iability Engineering Program, Department of Mechanical Engineering Historically, decision-makers have used expert opin i n to supplement lack of data. Expert opinion, however, is applied with much cauti on. This is because judgment is subjective and contains estimation error with so me degree of uncertainty. The purpose of this study is to quantify the uncert ainty surrounding the unknown of interest, given an expert opinion, in order to r educe the error of the estimate. This task is carried out by data-informed calibrati on and aggregation of expert opinion in a Bayesian framework. Additionally, thi s study evaluates the impact of the number of experts on the accuracy of aggregated estimate. The objective is to determine the correlation between the number of exp erts and the accuracy of the combined estimate in order to recommend an expert p anel size. DATA-INFORMED CALIBRATION AND AGGREGATION OF EXPERT OPINION IN A BAYESIAN FRAMEWORK

[1]  Roger M. Cooke,et al.  Fifteen years of expert judgement at TUDelft , 2008 .

[2]  Ronald Christensen,et al.  Bayesian Point Estimation Using the Predictive Distribution , 1985 .

[3]  A. Saltelli,et al.  Scenario and Parametric Uncertainty in GESAMAC. A Methodological Study in Nuclear Waste Disposal Risk Assessment. , 1999 .

[4]  Robert H. Ashton,et al.  Combining the judgments of experts: How many and which ones? , 1986 .

[5]  S. F. Klugman,et al.  Group Judgments for Familiar and Unfamiliar Materials , 1945 .

[6]  William F. Wright,et al.  Expertise and the explanation effect , 1988 .

[7]  W. Pike.  MODELING DRINKING WATER QUALITY VIOLATIONS WITH BAYESIAN NETWORKS 1 , 2004 .

[8]  Peter A. Morris,et al.  Observations on expert aggregation , 1986 .

[9]  Robert T. Clemen,et al.  Copula Models for Aggregating Expert Opinions , 1996, Oper. Res..

[10]  James H Vincent,et al.  Expert judgment and occupational hygiene: application to aerosol speciation in the nickel primary production industry. , 2003, The Annals of occupational hygiene.

[11]  K. A. Ericsson,et al.  Skilled memory and expertise: Mechanisms of exceptional performance. , 1988 .

[12]  Bill Ravens,et al.  An Introduction to Copulas , 2000, Technometrics.

[13]  James Mark Baldwin,et al.  Thought and things : a study of the development and meaning of thought or genetic logic , 1913 .

[14]  Nigel W. Arnell,et al.  Vulnerability to abrupt climate change in Europe , 2005 .

[15]  Bill Arnold,et al.  Modeling dispersion in three-dimensional heterogeneous fractured media at Yucca Mountain. , 2003, Journal of contaminant hydrology.

[16]  Bilal M. Ayyub,et al.  A Practical Guide on Conducting Expert-Opinion Elicitation of Probabilities and Consequences for Corps Facilities , 2001 .

[17]  P. Slovic.  Psychological Study of Human Judgment: Implications for Investment Decision-Making , 1972 .

[18]  A. H. Murphy,et al.  Reliability of Subjective Probability Forecasts of Precipitation and Temperature , 1977 .

[19]  J. Scott Armstrong,et al.  On the Selection of Error Measures for Comparisons Among Forecasting Methods , 2005 .

[20]  Bruce J. McLaren,et al.  Electric Bill Data , 2003 .

[21]  Bilal M. Ayyub,et al.  Book review of Elicitation of expert opinions for uncertainty and risks by Roger M. Cooke , 2003 .

[22]  John M Morris,et al.  Aggregating and Communicating Uncertainty. , 1980 .

[23]  Patricia M. Dechow,et al.  Returns to Contrarian Investment: Tests of the Naive Expectations Hypotheses , 1995 .

[24]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[25]  T. N. Krishnamurti,et al.  Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble. , 1999, Science.

[26]  Mike Metcalfe,et al.  Is Forecasting a Waste of Time? , 1996 .

[27]  M. Stone.  The Opinion Pool , 1961 .

[28]  Adrian K. Rantilla,et al.  Confidence in aggregation of expert opinions. , 2000, Acta psychologica.

[29]  A. Dawid.  The Well-Calibrated Bayesian , 1982 .

[30]  Paul R. Campbell,et al.  Evaluating Forecast Error in State Population Projections Using Census 2000 Counts , 2002 .

[31]  Baruch Fischhoff,et al.  Judgment under uncertainty: For those condemned to study the past: Heuristics and biases in hindsight , 1982 .

[32]  Nick T. Thomopoulos,et al.  Tables And Characteristics of the Standardized Lognormal Distribution , 2003 .

[33]  Robert T. Clemen,et al.  Future climate at Yucca Mountain, Nevada proposed high-level radioactive waste repository , 1995 .

[34]  Norman Crolee Dalkey,et al.  An experimental study of group opinion , 1969 .

[35]  H. A. Lingstone,et al.  The Delphi Method: Techniques and Applications , 1976 .

[36]  Robin M. Hogarth,et al.  A note on aggregating opinions , 1978 .

[37]  George Apostolakis,et al.  Some approximations useful to the use of dependent information sources , 1993 .

[38]  H. Kahn,et al.  The Year 2000: A Framework for Speculation on the Next Thirty-Three Years, , 1967 .

[39]  A Mark Williams,et al.  Perceptual-cognitive expertise in sport: some considerations when applying the expert performance approach. , 2005, Human movement science.

[40]  David C. Schmittlein,et al.  Combining Forecasts: Operational Adjustments to Theoretically Optimal Rules , 1990 .

[41]  Anthony O'Hagan,et al.  Eliciting expert beliefs in substantial practical applications , 1998 .

[42]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[43]  Anne Wingstrand,et al.  Differences and similarities among experts' opinions on Salmonella enterica dynamics in swine pre-harvest. , 2002, Preventive veterinary medicine.

[44]  M. J. Kallen,et al.  EXPERT AGGREGATION WITH DEPENDENCE , 2002 .

[45]  Victor Zarnowitz,et al.  Business Cycles Analysis and Expectational Survey Data , 1984 .

[46]  Enrico Zio,et al.  Two methods for the structured assessment of model uncertainty by experts in performance assessments of radioactive waste repositories , 1996 .

[47]  Roy Batchelor,et al.  Forecaster diversity and the benefits of combining forecasts , 1995 .

[48]  Roger M. Cooke,et al.  Uncertainty in dispersion and deposition in accident consequence modeling assessed with performance-based expert judgment , 1994 .

[49]  Kaplan,et al.  ‘Combining Probability Distributions from Experts in Risk Analysis’ , 2000, Risk analysis : an official publication of the Society for Risk Analysis.

[50]  Roger K. Blashfield,et al.  Performance of a composite as a function of the number of judges , 1978 .

[51]  Peter A. Morris,et al.  Combining Expert Judgments: A Bayesian Approach , 1977 .

[52]  Paul Slovic,et al.  From Shakespeare to Simon: speculations--and some evidence-- about man's ability to process information , 1972 .

[53]  Robert L. Winkler,et al.  Limits for the Precision and Value of Information from Dependent Sources , 1985, Oper. Res..

[54]  Christian Genest,et al.  Modeling Expert Judgments for Bayesian Updating , 1985 .

[55]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[56]  R. Clemen,et al.  Correlations and Copulas for Decision and Risk Analysis , 1999 .

[57]  Robert J. Crutcher,et al.  The role of deliberate practice in the acquisition of expert performance. , 1993 .

[58]  A O BaFail,et al.  APPLYING DATA MINING TECHNIQUES TO FORECAST NUMBER OF AIRLINE PASSENGERS IN SAUDI ARABIA (DOMESTIC AND INTERNATIONAL TRAVELS) , 2004 .

[59]  Christian Genest,et al.  Allocating the weights in the linear opinion pool , 1990 .

[60]  M. G. Morgan,et al.  Subjective judgments by climate experts. , 1995, Environmental science & technology.

[61]  Robert L. Winkler,et al.  Assessing Dependence: Some Experimental Results , 2000 .

[62]  J. van den Broeck,et al.  Influence of knowledge, training and experience of observers on the reliability of anthropometric measurements in children , 2003, Annals of human biology.

[63]  Ralph L. Keeney,et al.  On the uses of expert judgment on complex technical problems , 1989 .

[64]  M. Degroot.  A Bayesian view of assessing uncertainty and comparing expert opinion , 1988 .

[65]  T. Bedford,et al.  Expert elicitation for reliable system design , 2006, 0708.0279.

[66]  Rebecca A. Henry,et al.  Accuracy and confidence in group judgment , 1989 .

[67]  Gary D. Thompson,et al.  CARDON RESEARCH PAPERS IN AGRICULTURAL AND RESOURCE ECONOMICS , 2006 .

[68]  Doe.  SATURATED ZONE FLOW AND TRANSPORT EXPERT ELICITATION PROJECT , 1997 .

[69]  Yuhong Yang,et al.  Assessing Forecast Accuracy Measures , 2004 .

[70]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[71]  J. Charles Kerkering,et al.  Eliciting and Analyzing Expert Judgment, A Practical Guide , 2002, Technometrics.

[72]  R. L. Winkler,et al.  Separating probability elicitation from utilities , 1988 .

[73]  Ralph L. Keeney,et al.  Elicitation and use of expert judgment in performance assessment for high-level radioactive waste repositories , 1990 .

[74]  Mac McKee,et al.  Ionic and isotopic ratios for identification of salinity sources and missing data in the Gaza aquifer , 2006 .

[75]  Robert T. Clemen,et al.  The use of probability elicitation in the high-level nuclear waste regulation program , 1995 .

[76]  Fumika Ouchi,et al.  A literature review on the use of expert opinion in probabilistic risk analysis , 2004 .

[77]  Paul Slovic,et al.  Information processing, situation specificity, and the generality of risk-taking behavior , 1972 .

[78]  J. M. Bates,et al.  The Combination of Forecasts , 1969 .

[79]  R. E. Barlow,et al.  Introduction to de Finetti (1937) Foresight: Its Logical Laws, Its Subjective Sources , 1992 .

[80]  Mitchell J. Small,et al.  Expert System Methodology for Evaluating Reductive Dechlorination at TCE Sites , 1999 .

[81]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[82]  Hafiz T. A. Khan.  A comparative analysis of the accuracy of the United Nations population projections for six Southeast Asian countries , 2003 .

[83]  Robert H. Ashton,et al.  Aggregating Subjective Forecasts: Some Empirical Results , 1985 .

[84]  Robert B. Zajonc,et al.  A Note on Group Judgements and Group Size , 1962 .

[85]  Dino Gerardi,et al.  Aggregation of expert opinions , 2009, Games Econ. Behav..

[86]  J. Wilson,et al.  Network Representations of Knowledge about Chemical Equilibrium: Variations with Achievement. , 1994 .

[87]  Peter A. Morris,et al.  An Axiomatic Approach to Expert Resolution , 1983 .

[88]  Robert L. Winkler,et al.  The Consensus of Subjective Probability Distributions , 1968 .

[89]  J. Evans,et al.  Use of expert judgment in exposure assessment. Part I. Characterization of personal exposure to benzene , 2001, Journal of Exposure Analysis and Environmental Epidemiology.

[90]  Katherine D Walker,et al.  Use of expert judgment in exposure assessment: Part 2. Calibration of expert judgments about personal exposures to benzene , 2003, Journal of Exposure Analysis and Environmental Epidemiology.

[91]  Robert L. Winkler,et al.  Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit , 1993 .

[92]  Robert L. Winkler,et al.  Aggregating Point Estimates: A Flexible Modeling Approach , 1993 .

[93]  Thomas B. Sheridan,et al.  Filtering information from human experts , 1989, IEEE Trans. Syst. Man Cybern..

[94]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[95]  Yashika Forrester,et al.  THE QUALITY OF EXPERT JUDGMENT: AN INTERDISCIPLINARY INVESTIGATION , 2005 .

[96]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[97]  L H Goossens,et al.  Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. , 1998, Journal of radiological protection : official journal of the Society for Radiological Protection.

[98]  R. Ashton.  An Experimental Study of Internal Control Judgments , 1974 .

[99]  Robert Goodall Brown,et al.  Decision Making and Change in Human Affairs , 1979 .

[100]  Jeremy E. Oakley,et al.  Probability is perfect, but we can't elicit it perfectly , 2004, Reliab. Eng. Syst. Saf..

[101]  Eric Parent,et al.  Encoding prior experts judgments to improve risk analysis of extreme hydrological events via POT modeling , 2003 .

[102]  D. Winterfeldt,et al.  Nuclear waste and future societies: A look into the deep future , 1997 .

[103]  J. Hammitt,et al.  The Expected Value of Information and the Probability of Surprise , 1999 .

[104]  Ronald L. Iman,et al.  Expert opinion in risk analysis: the NUREG-1150 methodology , 1989 .

[105]  David A. Bessler,et al.  Forecasting Agricultural Prices Using a Bayesian Composite Approach , 1988, Journal of Agricultural and Applied Economics.

[106]  Herbert A. Simon,et al.  THE MIND'S EYE IN CHESS , 1988 .

[107]  Ali Mosleh,et al.  The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves , 1986 .

[108]  John S. Evans,et al.  Subjective Estimation of Toluene Exposures: A Calibration Study of Industrial Hygienists , 1989 .

[109]  Robin M. Hogarth,et al.  Quality of Group Judgment , 1977 .

[110]  Bobby E. Apostolakis.  Interfuel and energy-capital complementarity in manufacturing industries , 1990 .

[111]  Elizabeth C. Hirschman,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[112]  Z. Shipton,et al.  What do you think this is? "Conceptual uncertainty" in geoscience interpretation , 2007 .

[113]  Olaf Helmer,et al.  Problems in futures research: Delphi and causal cross-impact analysis , 1977 .

[114]  J. Kadane,et al.  Experiences in elicitation , 1998 .

[115]  D Chandramohan,et al.  Diagnostic accuracy of physician review, expert algorithms and data-derived algorithms in adult verbal autopsies. , 1999, International journal of epidemiology.

[116]  R. Hastie,et al.  Proper analysis of the accuracy of group judgments , 1997 .

[117]  Victor E. Grech,et al.  A comparison of clinical paediatric murmur assessment with echocardiography , 2004 .

[118]  J. G. Wissema,et al.  Trends in technology forecasting , 1982 .

[119]  Stephen Hora,et al.  Expert Panel Elicitation of Seismicity Following Glaciation in Sweden , 2005 .

[120]  Marina Pantazidou,et al.  Site-specific updating and aggregation of Bayesian belief network models for multiple experts. , 2004, Risk analysis : an official publication of the Society for Risk Analysis.

[121]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[122]  G. William Walster,et al.  A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups☆ , 1973 .

[123]  R. Cooke.  Elicitation of expert opinions for uncertainty and risks , 2003 .