Domain experts influence decision quality: Towards a robust method for their identification

Abstract Research indicates that practicing managers in the oil and gas industry invariably superimpose personal experience onto results of normative decision analysis tools. Decision quality therefore substantially depends upon the decision-makers level of expertise: an insufficient level, by implication, results in value destruction. Shanteau and colleagues have reported substantial work in the area of expertise, arguing that only people who are able to (a) differentiate between similar, but not identical, cases, and (b) repeat their judgment with consistency, qualify as ‘experts’. Methodologically, these researchers suggest a ratio of discrimination and inconsistency to distinguish experts. However, they acknowledge in their methodology it is possible (even for non-experts) to obtain inappropriate high scores using a consistent, but incorrect rule. While drawing inspiration from Shanteau's work, we suggest three important modifications based on our experiments, for use in the oil and gas industry: 1) Discrimination be defined as ability to discriminate correctly, as judged against a ‘gold standard’, not relative only to the cases presented, 2) Consistency be defined as consistent application of correct, not incorrect, rules, and 3) Inclusion of domain knowledge along with discrimination and consistency to assess expertise. Our initial experiments in distinguishing experts have been encouraging, even though results are only indicative given a small sample of twenty participants. Analysis shows expertise to be highly correlated with the breadth of experience (represented by basins or regions worked) and poorly with experience in the same geological environment (same basin, but multiple fields). The three cognitive abilities of domain knowledge, discrimination and consistency are also highly correlated with breadth of experience. Years of experience, our experiments confirm, is indeed an imperfect indicator of expertise.