Assessing the quality of expert judgment: Issues and analysis

Abstract Frequently the same biases have been manifest in experts as by students in the laboratory, but expertise studies are often no more ecologically valid than laboratory studies because the methods used in both are similar. Further, real-world tasks vary in their learnability, or the availability of outcome feedback necessary for a judge to improve performance with experience. We propose that good performance will be manifest when both ecological validity and learnability are high, but that performance will be poor when one of these is low. Finally, we suggest how researchers and practitioners might use these task-analytic constructs in order to identify true expertise for the formulation of decision support.

[1]  A. H. Murphy,et al.  Use of an Analogue Procedure to Formulate Objective Probabilistic Temperature Forecasts in The Netherlands , 1983 .

[2]  T D Koepsell,et al.  Effects of expertise and experience on risk judgments. , 1983, The Journal of applied psychology.

[3]  G. Northcraft,et al.  Experts, amateurs, and refrigerators: Comparing expert and amateur negotiators in a novel task , 1986 .

[4]  C R Peterson,et al.  Intuitive inference about normally distributed populations. , 1968, Journal of experimental psychology.

[5]  W. Edwards,et al.  Conservatism in a simple probability inference task. , 1966, Journal of experimental psychology.

[6]  P. Slovic,et al.  Value as a Determiner of Subjective Probability , 1966 .

[7]  Arthur E. Hoerl,et al.  Reliability of Subjective Evaluations in a High Incentive Situation , 1974 .

[8]  Wayne Snyder,et al.  Decision-Making with Risk and Uncertainty: The Case of Horse Racing , 1978 .

[9]  B. Fischhoff,et al.  Behavioral Decision Theory , 1977 .

[10]  Ranald R. Macdonald,et al.  More about linda or conjunctions in context , 1990 .

[11]  H J Steadman,et al.  Predicting dangerousness among the mentally ill: art, magic and science. , 1983, International journal of law and psychiatry.

[12]  Gordon F. Pitz,et al.  Decision Making and Cognition , 1977 .

[13]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[14]  R R Macdonald,et al.  Credible conceptions and implausible probabilities. , 1986, The British journal of mathematical and statistical psychology.

[15]  N. Anderson,et al.  Some psycholinguistic aspects of person perception , 1974, Memory & cognition.

[16]  E. Bamber,et al.  Expert Judgment in the Audit Team: A Source Reliability Approach , 1983 .

[17]  Stuart Oskamp,et al.  The relationship of clinical experience and training methods to several criteria of clinical prediction. , 1962 .

[18]  H. J. Einhorn.  Expert judgment: Some necessary conditions and an example. , 1974 .

[19]  T. Johnson.  Professions and Power , 1972 .

[20]  James Shanteau,et al.  Reducing the influence of irrelevant information on experienced decision makers , 1984 .

[21]  John W. Payne,et al.  Contingent decision behavior. , 1982 .

[22]  Peter Ayton,et al.  Immediate and short-term judgemental forecasting: Personologism, situationism or interactionism? , 1988 .

[23]  H. Raiffa,et al.  Applied Statistical Decision Theory. , 1961 .

[24]  P Manu,et al.  Judged frequency of complications after invasive diagnostic procedures. Systematic biases of a physician population. , 1984, Medical care.

[25]  G. Northcraft,et al.  Experts, amateurs, and real estate: An anchoring-and-adjustment perspective on property pricing decisions , 1987 .

[26]  P E Johnson,et al.  What kind of expert should a system be? , 1983, The Journal of medicine and philosophy.

[27]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[28]  A. H. Murphy,et al.  A comparative evaluation of objective and subjective weather forecasts in the united states , 1984 .

[29]  Hillel J. Einhorn,et al.  Judgment under uncertainty: Learning from experience and suboptimal rules in decision making , 1982 .

[30]  P. Elliott,et al.  The sociology of the professions , 1974 .

[31]  J. Edward Russo,et al.  An availability bias in professional judgment. , 1988 .

[32]  H. A. Wallace,et al.  What is in the Corn Judge's Mind?1 , 1923 .

[33]  W. Edwards.  The theory of decision making. , 1954, Psychological bulletin.

[34]  G. Pitz,et al.  Judgment and decision: theory and application. , 1984, Annual review of psychology.

[35]  James Shanteau,et al.  Psychological characteristics of expert decision makers , 1987 .

[36]  W A Wagenaar,et al.  Does the expert know? The reliability of predictions and confidence ratings of experts , 1986 .

[37]  Robert H. Ashton,et al.  Cue utilization and expert judgments: A comparison of independent auditors with other judges. , 1974 .

[38]  Peter Ayton,et al.  Judgemental probability forecasts for personal and impersonal events , 1989 .

[39]  W. Edwards,et al.  Decision Analysis and Behavioral Research , 1986 .

[40]  M. Riley,et al.  Conceptual competence and children's counting , 1984, Cognitive Psychology.

[41]  A. H. Murphy,et al.  Experiments in the laboratory and the real world , 1973 .

[42]  B. Fischhoff,et al.  Fault trees: Sensitivity of estimated failure probabilities to problem representation. , 1978 .

[43]  Cameron R. Peterson,et al.  Intuitive cascaded inferences , 1973 .

[44]  R. Edelmann,et al.  Mobility, social support, loneliness and well-being amongst two groups of older adults , 1988 .

[45]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[46]  P. Johnson-Laird,et al.  A theoretical analysis of insight into a reasoning task , 1970 .

[47]  Gideon Keren,et al.  Facing uncertainty in the game of bridge: A calibration study , 1987 .