Cognitive Processes and the Assessment of Subjective Probability Distributions

Abstract This article considers the implications of recent research on judgmental processes for the assessment of subjective probability distributions. It is argued that since man is a selective, sequential information processing system with limited capacity, he is ill-suited for assessing probability distributions. Various studies attesting to man's difficulties in acting as an “intuitive statistician” are summarized in support of this contention. The importance of task characteristics on judgmental performance is also emphasized. A critical survey of the probability assessment literature is provided and organized around five topics: (1) the “meaningfulness” of probability assessments; (2) methods of eliciting distributions; (3) feedback and evaluation of assessors; (4) differential ability of groups of assessors and (5) the problems of eliciting a single distribution from a group of assessors. Conclusions from the analysis with respect to future work include the need to capitalize on cognitive simplific...

[1]  T. Wallsten,et al.  Individual Decision Behavior , 1972 .

[2]  A. H. Murphy,et al.  Subjective Probability Forecasting in the Real World: Some Experimental Results , 1973 .

[3]  D. G. Pruitt,et al.  Informational requirements in making decisions. , 1961, The American journal of psychology.

[4]  John B. Kidd,et al.  The utilization of subjective probabilities in production planning , 1970 .

[5]  Paul Slovic,et al.  Manipulating the attractiveness of a gamble without changing its expected value. , 1969 .

[6]  Ward Edwards,et al.  Bayesian and regression models of human information processing—A myopic perspective , 1971 .

[7]  Lee Roy Beach,et al.  A note on the intrasubject similarity of subjective probabilities obtained by estimates and by bets , 1974 .

[8]  W. A. Wagenaar.  Appreciation of conditional probabilities in binary sequences , 1970 .

[9]  H. M. Jenkins,et al.  The display of information and the judgment of contingency. , 1965, Canadian journal of psychology.

[10]  JOHN COHEN,et al.  Evaluation of Compound Probabilities in Sequential Choice , 1971, Nature.

[11]  Lee Roy Beach,et al.  Data sequences and subjective sampling distributions , 1967 .

[12]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[13]  Carl-Axel S. Staël von Holstein,et al.  Two techniques for assessment of subjective probability distributions — An experimental study☆ , 1971 .

[14]  Robert Schlaifer,et al.  Computer programs for elementary decision analysis , 1971 .

[15]  K. R. Hammond,et al.  ANALYZING THE COMPONENTS OF CLINICAL INFERENCE. , 1964, Psychological review.

[16]  A. H. Murphy,et al.  forecasters and probability forecasts: some current problems1,2 , 1971 .

[17]  P. Slovic.  Psychological Study of Human Judgment: Implications for Investment Decision-Making , 1972 .

[18]  W. Edwards,et al.  Conservatism in a simple probability inference task. , 1966, Journal of experimental psychology.

[19]  David A. Schum,et al.  Concluding comments about the special issue on hierarchical inference , 1973 .

[20]  R. L. Winkler.  Scoring Rules and the Evaluation of Probability Assessors , 1969 .

[21]  George P. Huber,et al.  METHODS FOR QUANTIFYING SUBJECTIVE PROBABILITIES AND MULTI‐ATTRIBUTE UTILITIES*† , 1974 .

[22]  Dennis G. Fryback,et al.  Choices among bets by Las Vegas gamblers: Absolute and contextual effects , 1973 .

[23]  L. J. Chapman,et al.  Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. , 1969, Journal of abnormal psychology.

[24]  J. R. Newton.  JUDGMENT AND FEEDBACK IN A QUASI-CLINICAL SITUATION. , 1965, Journal of personality and social psychology.

[25]  John Cohen,et al.  Psychological Probability, Or, The Art Of Doubt , 1972 .

[26]  L. Beach,et al.  Man as an Intuitive Statistician , 2022 .

[27]  R. L. Winkler.  The Assessment of Prior Distributions in Bayesian Analysis , 1967 .

[28]  Norman Crolee Dalkey,et al.  An experimental study of group opinion , 1969 .

[29]  Ralf E. Schaefer,et al.  The assessment of subjective probability distributions: A training experiment☆ , 1973 .

[30]  Nico H. Frijda,et al.  Simulation of human long term memory , 1972 .

[31]  Lawrence D. Phillips,et al.  The ‘true probability’ problem , 1970 .

[32]  J. K. Clarkson,et al.  An explanation of conservatism in the bookbag-and-pokerchips situation , 1972 .

[33]  A. Tversky,et al.  BELIEF IN THE LAW OF SMALL NUMBERS , 1971, Pediatrics.

[34]  Michael L Donnell,et al.  Intrasubject comparison of four response modes for “subjective probability” assessment , 1973 .

[35]  N. Dalkey,et al.  The Delphi Method, III: Use of Self-Ratings To Improve Group Estimates , 1969 .

[36]  Robert L. Winkler,et al.  The Consensus of Subjective Probability Distributions , 1968 .

[37]  William K. Estes,et al.  Research and Theory on the Learning of Probabilities , 1972 .

[38]  H A SIMON,et al.  HUMAN ACQUISITION OF CONCEPTS FOR SEQUENTIAL PATTERNS. , 1963, Psychological review.

[39]  L. D. Pankoff,et al.  Bayesian synthesis of clinical and statistical prediction. , 1968 .

[40]  J. Davitz,et al.  A survey of studies contrasting the quality of group performance and individual performance, 1920-1957. , 1958, Psychological bulletin.

[41]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[42]  Hillel J. Einhorn,et al.  Expert measurement and mechanical combination , 1972 .

[43]  Donald M. Johnson,et al.  Systematic introduction to the psychology of thinking , 1972 .

[44]  A. H. Murphy,et al.  THE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON , 1970 .

[45]  L. R. Hoffman.  Group Problem Solving1 , 1965 .

[46]  Barbara C. Goodman,et al.  Action selection and likelihood ratio estimation by individuals and groups , 1972 .

[47]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[48]  J. K. Clarkson,et al.  Conservatism as non-Bayesian performance: A reply to De Swart , 1973 .

[49]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[50]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[51]  David H. Gustafson,et al.  Techniques for the Assessment of Worth , 1974 .

[52]  N. Anderson,et al.  Effect of stimulus inconsistency and discounting instructions in personality impression formation. , 1965, Journal of personality and social psychology.

[53]  James A. Wise,et al.  Subjective probability revision and subsequent decisions , 1969 .

[54]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[55]  N. Dalkey,et al.  An Experimental Application of the Delphi Method to the Use of Experts , 1963 .

[56]  Paul Slovic,et al.  Response-induced reversals of preference in gambling: An extended replication in las vegas , 1973 .

[57]  P. Meehl,et al.  Clinical versus Statistical Prediction. , 1955 .

[58]  David A. Schum,et al.  Research on human multistage probabilistic inference processes , 1973 .

[59]  Carl-Axel S. Staël von Holstein,et al.  The effect of learning on the assessment of subjective probability distributions , 1971 .

[60]  Maya Bar-Hillel,et al.  On the subjective probability of compound events , 1973 .

[61]  C. Hansel,et al.  The idea of a distribution. , 1955, British journal of psychology.

[62]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[63]  R. Graves.  The Greek Myths. Harmondsworth, Middlesex, England (Penguin Books) 1960. , 1960 .

[64]  S. Holstein,et al.  Assessment and evaluation of subjective probability distributions , 1970 .

[65]  Kenneth R. Hammond,et al.  Quasi-rationality, quarrels and new conceptions of feedback. , 1971 .

[66]  Lee Roy Beach,et al.  Intuitive statistical inferences about variances , 1968 .

[67]  Paul Slovic,et al.  From Shakespeare to Simon: speculations--and some evidence-- about man's ability to process information , 1972 .

[68]  A. H. Murphy,et al.  Scoring rules in probability assessment and evaluation , 1970 .

[69]  F. A. Jensen,et al.  Psychological effects of proper scoring rules , 1973 .

[70]  Lee Roy Beach,et al.  Equivalence Intervals and Decision Criteria , 1971 .

[71]  D. Magnusson,et al.  Cue relevance and feedback in a clinical prediction task , 1973 .

[72]  J. Spencer.  A FURTHER STUDY OF ESTIMATING AVERAGES , 1963 .

[73]  A. H. Murphy,et al.  credible interval temperature forecasts , 1972 .

[74]  M. Degroot.  Reaching a Consensus , 1974 .

[75]  B. deFinetti,et al.  METHODS FOR DISCRIMINATING LEVELS OF PARTIAL KNOWLEDGE CONCERNING A TEST ITEM. , 1965, The British journal of mathematical and statistical psychology.

[76]  J. March,et al.  A Behavioral Theory of the Firm , 1964 .

[77]  A. H. Murphy,et al.  Experiments in the laboratory and the real world , 1973 .

[78]  C. H. Coombs,et al.  Mathematical psychology : an elementary introduction , 1970 .

[79]  Charles A.J. Vlek,et al.  Learning probabilities of events: An analysis of the problem and its relevance for the study of decision making , 1970 .

[80]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[81]  J. Smedslund.  THE CONCEPT OF CORRELATION IN ADULTS , 1963 .

[82]  H. Thomas,et al.  Subjective Probability and its Measurement , 1973 .

[83]  Robert Schlaifer,et al.  Analysis of decisions under uncertainty , 1969 .

[84]  H. J. Einhorn.  Use of nonlinear, noncompensatory models as a function of task and amount of information , 1971 .

[85]  Robert L. Winkler,et al.  On the choice of a consensus distribution in Bayesian analysis , 1972 .

[86]  L. J. Chapman.  Illusory correlation in observational report. , 1967 .

[87]  Charles Vlek,et al.  Coherence of human judgment in a limited probabilistic environment , 1973 .

[88]  J. Bruner,et al.  A study of thinking , 1956 .

[89]  A. H. Murphy,et al.  Credible Interval Temperature Forecasting: Some Experimental Results , 1974 .

[90]  M. Stone.  The Opinion Pool , 1961 .

[91]  E. Eisenberg,et al.  CONSENSUS OF SUBJECTIVE PROBABILITIES: THE PARI-MUTUEL METHOD, , 1959 .

[92]  Gordon F. Pitz,et al.  On the processing of information: Probabilistic and otherwise , 1970 .

[93]  W. A. Wagenaar.  Generation of random sequences by human subjects: A critical survey of literature. , 1972 .

[94]  Wolfgang Manz,et al.  Experiments on probabilistic information processing , 1970 .

[95]  D. W. Martin,et al.  Feedback and response mode in performing a Bayesian decision task. , 1969, Journal of Applied Psychology.

[96]  A. Tversky,et al.  On the psychology of prediction , 1973 .

[97]  Cameron R. Peterson,et al.  Probabilistic information processing systems: Evaluation with conditionally dependent data☆ , 1972 .

[98]  F. Sanders,et al.  Skill In Forecasting Daily Temperature and Precipitation: Some Experimental Results , 1973 .

[99]  Ward Edwards,et al.  Probabilistic Information Processing Systems: Design and Evaluation , 1968, IEEE Trans. Syst. Sci. Cybern..

[100]  A. H. Murphy,et al.  Forecasters and Probability Forecasts: the Responses to a Questionnaire , 1971 .

[101]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[102]  James A. Wise,et al.  Subjective probability and decision strategy , 1969 .

[103]  H. Simon,et al.  Empirical tests of a theory of human acquisition of concepts for sequential patterns , 1973 .

[104]  H. M. Jenkins,et al.  JUDGMENT OF CONTINGENCY BETWEEN RESPONSES AND OUTCOMES. , 1965, Psychological monographs.

[105]  A. F. Sanders,et al.  Choice among bets and revision of opinion , 1968 .

[106]  F. Sanders.  On Subjective Probability Forecasting , 1963 .

[107]  T. A. Brown,et al.  An Experiment in Probabilistic Forecasting , 1973 .

[108]  Allen Newell,et al.  Human problem solving: The state of the theory in 1970. , 1971 .

[109]  Peter A. Morris,et al.  Decision Analysis Expert Use , 1974 .

[110]  G. William Walster,et al.  A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups☆ , 1973 .

[111]  Yuji Ijiri,et al.  Improving Subjective Probability Assessment for Planning and Control in Team-Like Organizations , 1974 .

[112]  C. Hansel,et al.  The idea of independence. , 1955, British journal of psychology.

[113]  Robert L. Winkler,et al.  An Introduction to Bayesian Inference and Decision , 1972 .

[114]  Floyd H. Allport,et al.  Theories of Perception and the Concept of Structure: A Review and Critical Analysis with an Introduction to a Dynamic-Structural Theory of Behavior. , 1955 .

[115]  A. H. Murphy,et al.  Nonlinear Utility and the Probability Score , 1970 .

[116]  Paul Slovic,et al.  Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment. , 1971 .

[117]  R M Hogarth,et al.  Process tracing in clinical judgment. , 1974, Behavioral science.

[118]  Kenneth R. Hammond,et al.  Negative effects of outcome-feedback in multiple-cue probability learning. , 1973 .

[119]  John Cohen,et al.  A CONFIRMATION OF THE INERTIAL‐ψ EFFECT IN SEQUENTIAL CHOICE AND DECISION , 1972 .

[120]  Barbara C Goodman,et al.  Direct Estimation Procedures for Eliciting Judgments About Uncertain Events , 1973 .

[121]  K R Hammond,et al.  Differential feedback in two multiple-cue probability learning tasks. , 1965, Behavioral science.

[122]  J. Sawyer,et al.  Measurement and prediction, clinical and statistical. , 1966, Psychological bulletin.

[123]  J. E. Matheson,et al.  The Elicitation of Continuous Probability Distributions , 1973 .

[124]  Harvey J. Brightman,et al.  The influence of the dogmatic personality upon information processing: A comparison with a Bayesian information processor , 1974 .

[125]  P Slovic,et al.  Cue-consistency and cue-utilization in judgment. , 1966, The American journal of psychology.

[126]  L D Phillips,et al.  Subjective propabilities inferred from estimates and bets. , 1967, Journal of experimental psychology.

[127]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[128]  John Cohen,et al.  THE DOCTRINE OF PSYCHOLOGICAL CHANCES , 1970 .

[129]  Howard Raiffa,et al.  Decision analysis: introductory lectures on choices under uncertainty. 1968. , 1969, M.D.Computing.

[130]  James Andrew Bartos,et al.  The assessment of probability distributions for future security prices , 1970 .

[131]  J. Smedslund,et al.  Note on learning, contingency, and clinical experience. , 1966, Scandinavian journal of psychology.

[132]  William L Sibley.  A Prototype Computer Program for Interactive Computer-Administered Admissible Probability Measurement , 1974 .

[133]  L. J. Savage,et al.  The Foundations of Statistics , 1955 .

[134]  E. H. Shuford,et al.  Admissible probability measurement procedures , 1966, Psychometrika.

[135]  R. L. Winkler.  The Quantification of Judgment: Some Methodological Suggestions , 1967 .

[136]  C. Holstein.  An Experiment in Probabilistic Weather Forecasting , 1971 .

[137]  Frederick Sanders.  The Verification of Probability Forecasts , 1967 .

[138]  E. Johnsen.  Richard M. Cyert & James G. March, A Behavioral Theory of The Firm, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, 1963, 332 s. , 1964 .

[139]  L. J. Chapman,et al.  Genesis of popular but erroneous psychodiagnostic observations. , 1967, Journal of abnormal psychology.

[140]  Mari Riess Jones,et al.  From probability learning to sequential processing: A critical review. , 1971 .

[141]  Dennis G. Fryback,et al.  Inferences based on unreliable reports , 1971 .

[142]  Joel S. Warm,et al.  Psychology of Perception , 1957 .

[143]  David H. Gustafson,et al.  Evaluation of probabilistic information processing in medical decision making , 1969 .

[144]  Carl-Axel S. Staël von Holstein,et al.  Probabilistic forecasting: An experiment related to the stock market , 1972 .

[145]  L. Beach,et al.  Intuitive estimation of means , 1966 .

[146]  Sarah Lichtenstein,et al.  The importance of the data-generating model in probability estimation☆ , 1968 .

[147]  B. D. Finetti.  La prévision : ses lois logiques, ses sources subjectives , 1937 .

[148]  G. Anthony Gorry,et al.  Automating Judgmental Decision Making for a Serious Medical Problem , 1971 .