A method for evaluating elicitation schemes for probabilistic models

We present an objective approach for evaluating probability and structure elicitation methods in probabilistic models. The main idea is to use the model derived from the experts' experience rather than the true model as the standard to compare the elicited model. We describe a general procedure by which it is possible to capture the data corresponding to the expert's beliefs, and we present a simple experiment in which we utilize this technique to compare three methods for eliciting discrete probabilities: 1) direct numerical assessment, 2) the probability wheel, and 3) the scaled probability bar. We show that for our domain, the scaled probability bar is the most effective tool for probability elicitation.

[1]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[2]  P. Spirtes,et al.  Causation, prediction, and search , 1993 .

[3]  Marek J. Druzdzel,et al.  SMILE: Structural Modeling, Inference, and Learning Engine and GeNIE: A Development Environment for Graphical Decision-Theoretic Models , 1999, AAAI/IAAI.

[4]  Silja Renooij,et al.  How to Elicit Many Probabilities , 1999, UAI.

[5]  David A. Schkade,et al.  WHERE DO THE NUMBERS COME FROM , 1993 .

[6]  Haiqin Wang,et al.  User Interface Tools for Navigation in Conditional Probability Tables and Elicitation of Probabilities in Bayesian Networks , 2000, UAI.

[7]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[8]  Marek J. Druzdzel,et al.  Building Probabilistic Networks: "Where Do the Numbers Come From?" Guest Editors Introduction , 2000, IEEE Trans. Knowl. Data Eng..

[9]  P. C. Price,et al.  Effects of a Relative-Frequency Elicitation Question on Likelihood Judgment Accuracy: The Case of External Correspondence. , 1998, Organizational behavior and human decision processes.

[10]  S. Hora,et al.  Assessment of probability distributions for continuous random variables: A comparison of the bisection and fixed value methods , 1992 .

[11]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[12]  David Heckerman,et al.  A Characterization of the Dirichlet Distribution with Application to Learning Bayesian Networks , 1995, UAI.

[13]  B. Fischhoff,et al.  Judged frequency of lethal events , 1978 .

[14]  W. Estes.  The cognitive side of probability learning. , 1976 .

[15]  R. Cooke.  Experts in Uncertainty: Opinion and Subjective Probability in Science , 1991 .

[16]  Gregory F. Cooper,et al.  A Bayesian Method for the Induction of Probabilistic Networks from Data , 1992 .

[17]  Judea Pearl,et al.  A Theory of Inferred Causation , 1991, KR.

[18]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[19]  M. W. Merkhofer,et al.  Quantifying judgmental uncertainty: Methodology, experiences, and insights , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[20]  Hanna Wasyluk,et al.  Extension of the HEPAR II Model to Multiple-Disorder Diagnosis , 2000, Intelligent Information Systems.

[21]  David Heckerman,et al.  Bayesian Networks for Data Mining , 2004, Data Mining and Knowledge Discovery.