Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.

[1]  Donald A. Waterman,et al.  A Guide to Expert Systems , 1986 .

[2]  Maria T. Correa,et al.  Effect of calfhood morbidity on age at first calving in New York Holstein herds , 1988 .

[3]  Roger M. Cooke,et al.  Applications of some risk assessment techniques: Formal expert judgement and accident sequence precursors , 1997 .

[4]  Ralph L. Keeney,et al.  Eliciting probabilities from experts in complex technical problems , 1991 .

[5]  Max Henrion,et al.  Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis , 1990 .

[6]  L H Goossens,et al.  Joint EC/USNRC expert judgement driven radiological protection uncertainty analysis. , 1998, Journal of radiological protection : official journal of the Society for Radiological Protection.

[7]  Stan Kaplan,et al.  ‘Expert information’ versus ‘expert opinions’. Another approach to the problem of eliciting/ combining/using expert knowledge in PRA , 1992 .

[8]  D. Forster,et al.  Predicting the number of herds infected with pseudorabies virus in the United States. , 1994, American journal of veterinary research.

[9]  Jon C. Helton,et al.  Probabilistic accident consequence uncertainty analysis: Dispersion and deposition uncertainty assessment, appendices A and B , 1995 .

[10]  Ralph L. Keeney,et al.  On the uses of expert judgment on complex technical problems , 1989 .

[11]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[12]  I R Dohoo,et al.  Calf and disease factors affecting growth in female Holstein calves in Florida, USA. , 1998, Preventive veterinary medicine.

[13]  A. W. Jalvingh,et al.  An economic model to calculate farm-specific losses due to bovine respiratory disease in dairy heifers. , 2001, Preventive veterinary medicine.

[14]  Jelle van Lenthe,et al.  The Development and Evaluation of ELI, an Interactive Elicitation Technique for Subjective Probability Distributions , 1993 .

[15]  R. Cooke,et al.  Procedures Guide for Structural Expert Judgement in Accident Consequence Modelling , 2000 .

[16]  D. C. Blood,et al.  Veterinary Medicine: A Textbook of the Diseases of Cattle, Sheep, Pigs, Goats and Horses , 1994 .

[17]  D Waltner-Toews,et al.  The effect of early calfhood health status on survivorship and age at first calving. , 1986, Canadian journal of veterinary research = Revue canadienne de recherche veterinaire.

[18]  H N Erb,et al.  The Effect of Calfhood Diseases on Growth of Female Dairy Calves During the First 3 Months of Life in New York State , 1996, Journal of Dairy Science.

[19]  A. A. Dijkhuizen,et al.  Introduction of contagious animal diseases into The Netherlands: elicitation of expert opinions , 1998 .

[20]  H N Erb,et al.  Lack of association between calf morbidity and subsequent first lactation milk production in 25 New York Holstein herds. , 1995, Journal of dairy science.

[21]  C. Allin Cornell,et al.  Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis * , 1998 .

[22]  Marija J. Norusis,et al.  SPSS for Windows Base System User''s Guide , 1992 .

[23]  H Otway,et al.  Expert judgment in risk analysis and management: process, context, and pitfalls. , 1992, Risk analysis : an official publication of the Society for Risk Analysis.

[24]  Jane M. Booker,et al.  Eliciting and analyzing expert judgement - a practical guide , 2001, ASA-SIAM series on statistics and applied probability.

[25]  Roger M. Cooke,et al.  Expert judgement and lethal toxicity of inhaled chemicals , 1998 .

[26]  Jelle van Lenthe,et al.  ELI : the use of proper scoring rules for eliciting subjective probability distributions , 1993 .

[27]  R. Huirne,et al.  Coping with Risk in Agriculture , 1997 .