Comparison of weighting judgments in multiattribute utility measurement

This paper compares four weighting methods in multiattribute utility measurement: the ratio method, the swing weighting method, the tradeoff method and the pricing out method. 200 subjects used these methods to weight attributes for evaluating nuclear waste repository sites in the United States. The weighting methods were compared with respect to their internal consistency, convergent validity, and external validity. Internal consistency was measured by the degree to which ordinal and cardinal or ratio responses agreed within the same weighting method. Convergent validity was measured by the degree of agreement between the weights elicited with different methods. External validity was determined by the degree to which weights elicited in this experiment agreed with weights that were elicited with managers of the Department of Energy. In terms of internal consistency, the tradeoff method fared worst. In terms of convergent validity, the pricing out method turned out to be an outlier. In terms of external validity, the pricing out method showed the best results. While the ratio and swing methods are quite consistent and show a fair amount of convergent validity, their external validity problems cast doubt on their usefulness. The main recommendation for applications is to improve the internal consistency of the tradeoff method by careful interactive elicitation and to use it in conjunction with the pricing out method to enhance its external validity.

[1]  D. Winterfeldt,et al.  The effect of varying value trees on multiattribute evaluations , 1988 .

[2]  A. Tversky,et al.  Contingent weighting in judgment and choice , 1988 .

[3]  R L Keeney,et al.  A multiattribute utility analysis of alternative sites for the disposal of nuclear waste. , 1987, Risk analysis : an official publication of the Society for Risk Analysis.

[4]  D. Winterfeldt,et al.  Comparing Hierarchical and Nonhierarchical Weighting Methods for Eliciting Multiattribute Value Models , 1987 .

[5]  W. Edwards,et al.  Decision Analysis and Behavioral Research , 1986 .

[6]  D. Brinberg,et al.  Assessing attribute importance: a comparison of six methods , 1986 .

[7]  Leonard Adelman,et al.  The role of task properties in determining the relative effectiveness of multiattribute weighting techniques , 1984 .

[8]  Ward Edwards,et al.  Evaluating credit applications: A validation of multiattribute utility weight elicitation techniques , 1983 .

[9]  C. C. Waid,et al.  An Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models , 1982 .

[10]  R. John,et al.  A criterion validation of multiattribute utility analysis and of group communication strategy , 1980 .

[11]  K. Aschenbrenner,et al.  Challenging the cushing syndrome: Multiattribute evaluation of cortisone drugs , 1978 .

[12]  D. E. Bell,et al.  Conflicting Objectives in Decisions , 1978 .

[13]  G. W. Fischer,et al.  Convergent validation of decomposed multi-attribute utility assessment procedures for risky and riskless decisions , 1977 .

[14]  G. W. Fischer,et al.  Multidimensional utility models for risky and riskless choice , 1976 .

[15]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[16]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[17]  Robert T. Eckenrode,et al.  Weighting Multiple Criteria , 1965 .