Performance-based assessment of expertise: How to decide if someone is an expert or not

Abstract The identification of an expert is vital to any study or application involving expertise. If external criterion (a “gold standard”) exists, then identification is straightforward: Simply compare people against the standard and select whoever is closest. However, such criteria are seldom available for domains where experts work; that's why experts are needed in the first place. The purpose here is to explore various methods for identifying experts in the absence of a gold standard. One particularly promising approach (labeled CWS for Cochran–Weiss–Shanteau ) is explored in detail. We illustrate CWS through reanalyses of three previous studies of experts. In each case, CWS provided new insights into identifying experts. When applied to auditors, CWS correctly detected group differences in expertise. For agricultural judges, CWS revealed subtle distinctions between subspecialties of experts. In personnel selection, CWS showed that irrelevant attributes were more informative than relevant attributes. We believe CWS provides a valuable tool for identification and evaluation of experts.

[1]  J. Shanteau,et al.  Livestock judges: How much information can an expert use? , 1978 .

[2]  E. Salas,et al.  Linking Expertise and Naturalistic Decision Making , 2001 .

[3]  H. J. Einhorn.  Expert judgment: Some necessary conditions and an example. , 1974 .

[4]  Thomas Kida,et al.  The Effect Of Causality And Specificity On Data Use , 1984 .

[5]  L. R. Goldberg.  Simple models or simple processes? Some research on clinical judgments. , 1968, The American psychologist.

[6]  A. H. Ashton,et al.  Does consensus imply accuracy in accounting studies of decision making?: Alison Hubbard Ashton, Accounting Review 60 (185) 173–85 , 1986 .

[7]  D. Lykken,et al.  The detection of deception. , 1979, Psychological bulletin.

[8]  P. Slovic.  Analyzing the expert judge: A descriptive study of a stockbroker's decision process. , 1969 .

[9]  W. G. Cochran.  The Comparison of Different Scales of Measurement for Experimental Results , 1943 .

[10]  R. A. Bradley,et al.  THE COMPARISON OF THE SENSITIVITIES OF SIMILAR EXPERIMENTS. MODEL II OF THE ANALYSIS OF VARIANCE , 1959 .

[11]  J. Shanteau.  Psychological characteristics and strategies of expert decision makers , 1988 .

[12]  M. Chi.  Knowledge structures and memory development. , 1978 .

[13]  James Shanteau,et al.  Decision Making by Experts: The GNAHM Effect , 1999 .

[14]  K. R. Hammond.  Human judgment and social policy , 1980 .

[15]  C E Werts,et al.  The reliability of clinician's judgments: a multitrait-multimethod approach. , 1966, Journal of consulting psychology.

[16]  Barbara A. Mellers,et al.  Decision science and technology. Reflections on the contributions of Ward Edwards , 1999 .

[17]  Hillel J. Einhorn,et al.  Expert measurement and mechanical combination , 1972 .

[18]  D. C. Raskin,et al.  Truth and deception: a reply to Lykken. , 1979, Psychological bulletin.

[19]  David J. Weiss,et al.  Schubrad: The comparison of the sensitivities of similar experiments , 1985 .

[20]  James Shanteau,et al.  Personal attributes of expert auditors , 1992 .

[21]  P. Todd,et al.  Simple Heuristics That Make Us Smart , 1999 .

[22]  I. Janis.  Victims Of Groupthink , 1972 .

[23]  T. R. Stewart,et al.  The Importance of the Task in Analyzing Expert Judgment , 1997 .