Experimental Tests of Subjective Bayesian Methods

We evaluated Samaniego and Reneau’s 1994 novel weight method for eliciting subjective probability estimates. Experiment 1 replicated their experiment (subjects weighed their prior estimate against 10 new observations), with an additional weight judgment against 50 observations. In Experiment 2, subjects gave prior estimates to questions in a domain more familiar to them. In Experiment 3, subjects weighed priors against samples of 10 or 50 on a modified scale. We find substantial individual differences in sensitivity to sample size in this task, and also evidence for implicit learning: Subjects who first judged N = 50 subsequently place lower weights on N = 10. People cannot evaluate the quality of their prior knowledge accurately, nor can they consistently produce calibrated priors and marginal posteriors. These results do not support the use of the weight-judgment method for parameter estimation, but do encourage its continued development and improvement.

[1]  Robin M. Hogarth,et al.  Cognitive Processes and the Assessment of Subjective Probability Distributions , 1975 .

[2]  A. H. Murphy,et al.  CAN WEATHER FORECASTERS FORMULATE RELIABLE PROBABILITY FORECASTS OF PRECIPITATION AND TEMPERATURES ? , 1977 .

[3]  G. W. Fischer,et al.  Scoring-rule feedback and the overconfidence syndrome in subjective probability forecasting , 1982 .

[4]  F. Samaniego,et al.  Toward a Reconciliation of the Bayesian and Frequentist Approaches to Point Estimation , 1994 .

[5]  Dean P. Foster,et al.  Precision and Accuracy of Judgmental Estimation , 1997 .

[6]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[7]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[8]  Jelle van Lenthe,et al.  Scoring-Rule Feedforward and the Elicitation of Subjective Probability Distributions , 1994 .

[9]  Arthur E. Hoerl,et al.  Reliability of Subjective Evaluations in a High Incentive Situation , 1974 .

[10]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[11]  H. Arkes,et al.  Two methods of reducing overconfidence , 1987 .

[12]  M. R. Novick,et al.  Statistical methods for educational and psychological research , 1976 .

[13]  Gideon Keren,et al.  On The Calibration of Probability Judgments: Some Critical Comments and Alternative Perspectives , 1997 .

[14]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[15]  Mark T. Spence.  Problem-Problem Solver Characteristics Affectingthe Calibration of Judgments , 1996 .

[16]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[17]  Jay J.J. Christensen-Szalanski,et al.  Physicians' use of probabilistic information in a real clinical setting. , 1981 .

[18]  A. Tversky,et al.  The weighing of evidence and the determinants of confidence , 1992, Cognitive Psychology.

[19]  D. Krantz,et al.  Mathematics Anxiety and Science Careers among Able College Women , 1992 .

[20]  B. Fischhoff,et al.  Reasons for confidence. , 1980 .

[21]  Ayleen Wisudha,et al.  Distribution of probability assessments for almanac and future event questions , 1982 .

[22]  W. Maugham.  Of Human Bondage , 1998, Annals of Internal Medicine.

[23]  S. Oskamp.  OVERCONFIDENCE IN CASE-STUDY JUDGMENTS. , 1965, Journal of consulting psychology.

[24]  Baruch Fischhoff,et al.  Focusing Techniques: A Shortcut to Improving Probability Judgments? , 1984 .

[25]  Roger Cooke,et al.  Calibration and information in expert resolution; a classical approach , 1988, Autom..