The Development and Evaluation of ELI, an Interactive Elicitation Technique for Subjective Probability Distributions

Uncertain knowledge about continuous quantities is usually formalized through Subjective Probability Distributions (SPD’s). Past research showed that the quality of SPD’s is rather poor and that poor SPD quality might originate from method-induced biases. It is quite possible that equipped with more appropriate elicitation tools, assessors will prove to be more competent probability estimators than research thus far suggested. The present paper describes the search for a new elicitation methodology. The most important feature of the new ELIcitation technique ELI is the direct realization of a proper scoring rule in a graphically oriented interactive Computer program. An evaluation study in which ELI was compared with other elicitation techniques showed that ELI Performance was superior.

[1]  R. E. Schaefer,et al.  The evaluation of individual and aggregated subjective probability distributions , 1976 .

[2]  J. B. Wallace,et al.  A STUDY OF THE PERFORMANCE OF SUBJECTIVE PROBABILITY ASSESSORS , 1974 .

[3]  David L. Ronis,et al.  Components of probability judgment accuracy: Individual consistency and effects of subject matter and assessment method. , 1987 .

[4]  D. A. Seaver,et al.  Eliciting subjective probability distributions on continuous variables , 1978 .

[5]  Ayleen Wisudha,et al.  Distribution of probability assessments for almanac and future event questions , 1982 .

[6]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[7]  Roger Cooke,et al.  Calibration and information in expert resolution; a classical approach , 1988, Autom..

[8]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[9]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[10]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[11]  D H Gustafson,et al.  Comparison of five methods for estimating subjective probability distributions. , 1977, Organizational behavior and human performance.

[12]  Robert F. Van Naerssen,et al.  A scale for the measurement of subjective probability , 1962 .

[13]  Carl-Axel S. Staël von Holstein,et al.  Measurement of subjective probability , 1970 .

[14]  David L. Ronis,et al.  Probability judgment accuracy: China, Japan, and the United States , 1989 .

[15]  Carl-Axel S. Staël von Holstein,et al.  Two techniques for assessment of subjective probability distributions — An experimental study☆ , 1971 .

[16]  George P. Huber,et al.  METHODS FOR QUANTIFYING SUBJECTIVE PROBABILITIES AND MULTI‐ATTRIBUTE UTILITIES*† , 1974 .

[17]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[18]  David V. Budescu,et al.  Encoding subjective probabilities: A psychological and psychometric review , 1983 .

[19]  Robin M. Hogarth,et al.  Cognitive Processes and the Assessment of Subjective Probability Distributions , 1975 .

[20]  G. W. Fischer,et al.  Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting , 1982 .

[21]  Herbert Moskowitz,et al.  Modified PERT versus fractile assessment of subjective probability distributions , 1979 .

[22]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[23]  R. Hogarth.  Judgement and choice: The psychology of decision , 1982 .

[24]  M. R. Novick,et al.  Statistical methods for educational and psychological research , 1976 .

[25]  W. Edwards,et al.  Decision Analysis and Behavioral Research , 1986 .

[26]  Jelle van Lenthe,et al.  ELI: An Interactive Elicitation Technique for Subjective Probability Distributions , 1993 .

[27]  A. H. Murphy,et al.  Scoring rules in probability assessment and evaluation , 1970 .