The process-performance paradox in expert judgment 197 2

A mysterious fatal disease strikes a large minority of the population. The disease is incurable, but an expensive drug can keep victims alive. Con ... gress decides that the drug should be given to those whose lives can be extended longest, which only a few specialists can predict. The experts work around the clock searchlng for a cure; allocating the drug is a new chore they would rather avoid. . In research on decision making there are two views about such experts. The views suggest different techno]ogies for modeJing experts' decisions so that they can do productive research rather than make predictions. One view, which emerges from behavioral research on decision making, is skeptical about the experts. Data suggest that a wide range of experts like our hypothetica1 specialists are not much better predictors than Jess expert physicians, or interns. Furthennore, this view suggests a simple technology for replacing experts a simple linear regression model (perhaps using medica] judgments as inputs). The regression does not mimic the thought process of an expert, but it proDa bJy makes more accurate predictions than an expert does. The second view, stemming from research in cognitive science, suggests that expertise is a rare skill that develops only after much instruction, practice, and experience. The cognition of experts is more sophisticated than that of novices; this sophistication is presumed to produce better predictions. This view suggests a model that strives to mimic the decision policies of experts an "expert (or knowledge-based) system" containing lists of rules experts use in judging longevity. An expert system tries to match, not exceed, the performance of the expert it represents. In this chapter we describe and integrate these two perspectives. Integration comes from realizing that the behavioral and cognitive science approaches have different goals: Whereas behavioral decision theory emphasizes the performance of experts, cognitive science usually emphasizes differences in experts' processes (E. Johnson, 1988). A few caveats are appropriate. Our review is selective; it is meant to emphasize the differences between expert performance and process. The generic

[1]  S. B. Levenberg.  Professional training, psychodiagnostic skill, and kinetic family drawings. , 1975, Journal of personality assessment.

[2]  J. Shanteau.  Psychological characteristics and strategies of expert decision makers , 1988 .

[3]  H L Kundel,et al.  Visual search patterns and experience with radiological images. , 1972, Radiology.

[4]  T. Sarbin.  The logic of prediction in psychology. , 1944 .

[5]  R. Cattell,et al.  Formal representation of human judgment , 1968 .

[6]  John H. Holland,et al.  Induction: Processes of Inference, Learning, and Discovery , 1987, IEEE Expert.

[7]  Jay J.J. Christensen-Szalanski,et al.  Physicians' use of probabilistic information in a real clinical setting. , 1981 .

[8]  H. J. Einhorn,et al.  Linear regression and process-tracing models of judgment. , 1979 .

[9]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[10]  John W. Payne,et al.  Effort and Accuracy in Choice , 1985 .

[11]  J. Sawyer,et al.  Measurement and prediction, clinical and statistical. , 1966, Psychological bulletin.

[12]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[13]  A. Tversky,et al.  Judgments of and by Representativeness , 1981 .

[14]  Colin F. Camerer,et al.  General conditions for the success of bootstrapping models , 1981 .

[15]  B. Brehmer.  In one word: Not from experience. , 1980 .

[16]  W. Chase.  Spatial Representations of Taxi Drivers , 1983 .

[17]  Theodore J. Mock,et al.  Expert and novice problem solving behaviour in audit planning: An experimental study , 1990 .

[18]  T. Pearson,et al.  Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[19]  Robert J. Meyer,et al.  The Learning of Multiattribute Judgment Policies , 1987 .

[20]  K. R. Hammond.  Probabilistic functioning and the clinical method. , 1955, Psychological review.

[21]  David L. Post,et al.  Medical Problem Solving: An Analysis of Clinical Reasoning , 1979 .

[22]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[23]  R. Dawes,et al.  Heuristics and Biases: Clinical versus Actuarial Judgment , 2002 .

[24]  L. R. Goldberg,et al.  The effectiveness of clinicians' judgments; the diagnosis of organic brain damage from the Bender-Gestalt test. , 1959, Journal of consulting psychology.

[25]  L. J. Chapman,et al.  Genesis of popular but erroneous psychodiagnostic observations. , 1967, Journal of abnormal psychology.

[26]  H. Simon,et al.  Perception in chess , 1973 .

[27]  E. L. Kelly.  Clinical versus statistical prediction: A theoretical analysis and review of the evidence. , 1955 .

[28]  Michael I. Jordan.  An introduction to linear algebra in parallel distributed processing , 1986 .

[29]  Hans J. Berliner,et al.  Backgammon Computer Program Beats World Champion , 1980 .

[30]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[31]  Peter G. Polson,et al.  An experimental analysis of the mechanisms of a memory skill. , 1988 .

[32]  Hillel J. Einhorn,et al.  Expert measurement and mechanical combination , 1972 .

[33]  Kenneth R. Hammond.  Toward a unified approach to the study of expert judgment , 1987 .

[34]  E. H. Bowman.  Consistency and Optimality in Managerial Decision Making , 1963 .

[35]  H. J. Einhorn,et al.  Accepting error to make less error. , 1986, Journal of personality assessment.

[36]  P. Meehl.  Causes and effects of my disturbing little book. , 1986, Journal of personality assessment.

[37]  Lewis R. Goldberg,et al.  Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. , 1970 .

[38]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[39]  H. Garb,et al.  Clinical judgment, clinical training, and professional experience. , 1989, Psychological bulletin.

[40]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[41]  Jean Bédard,et al.  Expertise in auditing: Myth or reality? , 1989 .

[42]  Willem A. Wagenaar,et al.  Calibration of probability assessments by professional blackjack dealers, statistical experts, and lay people , 1985 .

[43]  Eric J. Johnson,et al.  Expertise and decision under uncertainty: Performance and process. , 1988 .

[44]  M. Brucks.  The Effects of Product Class Knowledge on Information Search Behavior , 1985 .

[45]  J. Klayman,et al.  Confirmation, Disconfirmation, and Informa-tion in Hypothesis Testing , 1987 .

[46]  L. J. Chapman,et al.  Illusory correlation as an obstacle to the use of valid psychodiagnostic signs. , 1969, Journal of abnormal psychology.

[47]  Dorothea P. Simon,et al.  Expert and Novice Performance in Solving Physics Problems , 1980, Science.

[48]  N. Wiggins,et al.  Man versus model of man revisited: The forecasting of graduate school success. , 1971 .

[49]  Gideon Keren,et al.  Facing uncertainty in the game of bridge: A calibration study , 1987 .

[50]  W G Chase,et al.  Exceptional memory. , 1982, American scientist.

[51]  J. Klayman.  Cue discovery in probabilistic environments: Uncertainty and experimentation. , 1988 .

[52]  H. J. Einhorn.  Expert judgment: Some necessary conditions and an example. , 1974 .

[53]  J R Thornbury,et al.  A second look at the utility of radiographic skull examination for trauma. , 1979, AJR. American journal of roentgenology.

[54]  Robert Libby,et al.  Experience And The Ability To Explain Audit Findings , 1990 .

[55]  Robert Libby,et al.  Man versus model of man: some conflicting evidence , 1976 .

[56]  Eric J. Johnson,et al.  Product familiarity and learning new information , 1984 .

[57]  A. H. Murphy,et al.  CAN WEATHER FORECASTERS FORMULATE RELIABLE PROBABILITY FORECASTS OF PRECIPITATION AND TEMPERATURES ? , 1977 .

[58]  P. E. Johnson,et al.  Multimethod study of clinical judgment. , 1982, Organizational behavior and human performance.

[59]  Timothy A. Post,et al.  On the solving of ill-structured problems. , 1988 .

[60]  B Carroll,et al.  Artificial intelligence. Expert systems for clinical diagnosis: are they worth the effort? , 1987, Behavioral science.

[61]  R. Dawes.  A case study of graduate admissions: Application of three principles of human decision making. , 1971 .

[62]  D. B. Yntema,et al.  Man-Computer Cooperation in Decisions Requiring Common Sense , 1961 .

[63]  Daniel Kahneman,et al.  Availability: A heuristic for judging frequency and probability , 1973 .

[64]  P. Hoffman.  The paramorphic representation of clinical judgment. , 1960, Psychological bulletin.