Subjective Probability and Information Retrieval: a Review of the Psychological literature

The psychological literature on subjective probability estimation is reviewed to determine the feasibility of designing probabilistic information retrieval systems using such estimates. Their use has been considered by some writers, but psychological issues have not been addressed. Research pertinent to probabilistic information retrieval is examined and implications for probabilistic information retrieval are drawn. It is shown that accurate human probability estimation is possible, both in the laboratory and in real world tasks, e.g., in meteorological forecasting; but that it is also a task subject to systematic bias, or inaccuracy. Proposed techniques for debiasing are considered. The highly task‐dependent nature of such estimates is also discussed; two implications are that results from laboratory studies may have limited relevance to real world tasks and that empirical studies specific to the context of information retrieval need to be made. Human probability estimation appears to be a difficult task, but one which can be done well with proper training and use of debiasing techniques. It is premature to say how useful such estimates would be in probabilistic information retrieval, but their use should not yet be ruled out.

[1]  Hubert L. Dreyfus,et al.  Mind over Machine: The Power of Human Intuition and Expertise in the Era of the Computer , 1987, IEEE Expert.

[2]  H. Raiffa,et al.  Judgment under uncertainty: A progress report on the training of probability assessors , 1982 .

[3]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[4]  David Lindley.  Scoring rules and the inevitability of probability , 1982 .

[5]  Ward Edwards,et al.  Bayesian statistical inference for psychological research. , 1963 .

[6]  Donald H. Kraft,et al.  Operations Research Applied to Document Indexing and Retrieval Decisions , 1977, JACM.

[7]  Christine L. Borgman,et al.  The user's mental model of an information retrieval system , 1999, SIGIR '85.

[8]  Baruch Fischhoff,et al.  Calibrating databases , 1986, J. Am. Soc. Inf. Sci..

[9]  S. Bradley,et al.  A Dynamic Model for Bond Portfolio Management , 1972 .

[10]  Robert F. Nau,et al.  Coherent Assessment of Subjective Probability , 1981 .

[11]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[12]  F. Heider.  The psychology of interpersonal relations , 1958 .

[13]  D. Titterington,et al.  Comparison of Discrimination Techniques Applied to a Complex Data Set of Head Injured Patients , 1981 .

[14]  Izak Benbasat,et al.  Behavioral Aspects of Information Processing for the Design of Management Information Systems , 1982, IEEE Transactions on Systems, Man, and Cybernetics.

[15]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[16]  B. Fischhoff,et al.  Behavioral Decision Theory , 1977 .

[17]  D H Gustafson,et al.  Probabilistic information processing: implementation and evaluation of a Semi-PIP diagnostic system. , 1973, Computers and biomedical research, an international journal.

[18]  Paul Slovic,et al.  Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment. , 1971 .

[19]  David H. Gustafson,et al.  Evaluation of probabilistic information processing in medical decision making , 1969 .

[20]  W. Edwards.  The theory of decision making. , 1954, Psychological bulletin.

[21]  G. Pitz,et al.  Judgment and decision: theory and application. , 1984, Annual review of psychology.

[22]  Carl-Axel S. Staël von Holstein,et al.  Exceptional Paper---Probability Encoding in Decision Analysis , 1975 .

[23]  B. H. Beach.  Expert judgment about uncertainty: Bayesian decision making in realistic settings. , 1975, Organizational behavior and human performance.

[24]  A. Tversky,et al.  Prospect Theory : An Analysis of Decision under Risk Author ( s ) : , 2007 .

[25]  D. Berkeley,et al.  Structuring decision problems and the ‘bias heuristic’ , 1982 .

[26]  D. Cox.  Two further applications of a model for binary regression , 1958 .

[27]  David Navon.  Statistical and metastatistical considerations in analysing the desirability of human Bayesian conservatism , 1981 .

[28]  Paul Slovic,et al.  Multidimensional functional learning (MFL) and some new conceptions of feedback , 1981 .

[29]  Christine L. Borgman.  End user behavior on an online information retrieval system: a computer monitoring study , 1983, SIGIR 1983.

[30]  Sverre Petterssen,et al.  Weather analysis and forecasting , 1940 .

[31]  L. Beach,et al.  Man as an Intuitive Statistician , 2022 .

[32]  Wayne S. Smith,et al.  Interactive Elicitation of Opinion for a Normal Linear Model , 1980 .

[33]  William S. Cooper,et al.  Indexing documents by gedanken experimentation , 1978, J. Am. Soc. Inf. Sci..

[34]  A. H. Murphy,et al.  Experiments in the laboratory and the real world , 1973 .

[35]  Jean Tague,et al.  A Bayesian approach to interactive retrieval , 1973 .

[36]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[37]  Ward Edwards,et al.  Dynamic Decision Theory and Probabilistic Information Processings1 , 1962 .

[38]  R. Hogarth.  Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. , 1981 .

[39]  William C. Howell,et al.  A test of task influences in uncertainty measurement , 1982 .

[40]  G. Salton,et al.  A Generalized Term Dependence Model in Information Retrieval , 1983 .

[41]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[42]  David V. Budescu,et al.  Encoding subjective probabilities: A psychological and psychometric review , 1983 .

[43]  B. Fischhoff,et al.  Hypothesis Evaluation from a Bayesian Perspective. , 1983 .

[44]  Ian I. Mitroff,et al.  A Program for Research on Management Information Systems , 1973 .

[45]  Don R. Swanson,et al.  A decision theoretic foundation for indexing , 1975, J. Am. Soc. Inf. Sci..

[46]  Cameron R. Peterson,et al.  Probabilistic information processing systems: Evaluation with conditionally dependent data☆ , 1972 .

[47]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[48]  N. Anderson.  Foundations of information integration theory , 1981 .

[49]  A. Tversky,et al.  Prospect theory: analysis of decision under risk , 1979 .

[50]  A. Tversky,et al.  Judgments of and by Representativeness , 1981 .

[51]  Alan F. Smeaton,et al.  The Retrieval Effects of Query Expansion on a Feedback Document Retrieval System , 1983, Comput. J..

[52]  M. E. Maron,et al.  On indexing, retrieval and the meaning of about , 1977, J. Am. Soc. Inf. Sci..

[53]  William F. Wright.  Properties of judgment models in a financial setting , 1979 .

[54]  D. Navon.  The importance of being conservative: Some reflections on human Bayesian behaviour , 1978 .

[55]  A. Tversky,et al.  Intuitive Prediction: Biases and Corrective Procedures , 1982 .

[56]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[57]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[58]  B. D. Finetti,et al.  Probability, induction and statistics : the art of guessing , 1979 .

[59]  A. Tversky,et al.  On the Reconciliation of Probability Assessments , 1979 .