A comparison of three weight elicitation methods: good, better, and best

This paper compares the properties and performance of three weight elicitation methods. It is in effect a "second round contest" in which the Bottomley et al. (2000) champion, direct rating (DR), locks horns with two new challengers. People using DR rate each attribute in turn on a scale of 0-100, whilst people using Max100 first assign to the most important attribute(s) a rating of 100, and then rate the other attributes relative to it/them. People using Min10 first assign the least important attribute(s) a rating of 10, and then rate the other attributes relative to it/them. The weights produced by Max100 were somewhat more test-retest reliable than DR. Both methods were considerably more reliable than Min10. Using people's test-retest data as attribute weights on simulated alternative values in a multi-attribute choice scenario, the same alternative would be chosen on 91% of occasions using Max100, 87% of occasions using DR, but only 75% of occasions using Min10. Moreover, the three methods are shown to have very distinct "signatures", that is profiles relating weights to rank position. Finally, people actually preferred using Max100 and DR rather than Min10, an important pragmatic consideration.

[1]  R. Green,et al.  Judging Relative Importance: Direct Rating and Point Allocation Are Not Equivalent , 1997, Organizational behavior and human decision processes.

[2]  H. J. Einhorn,et al.  Expression theory and the preference reversal phenomena. , 1987 .

[3]  John W. Tukey,et al.  Exploratory Data Analysis. , 1979 .

[4]  W. Edwards,et al.  Decision Analysis and Behavioral Research , 1986 .

[5]  JAN-BENEDICT STEENKAMP,et al.  Attribute Elicitation in Marketing Research: A Comparison of Three Procedures , 1997 .

[6]  Robert T. Eckenrode,et al.  Weighting Multiple Criteria , 1965 .

[7]  D. Budescu,et al.  A Psychometric Analysis of the "Divide and Conquer" Principle in Multicriteria Decision Making. , 1998, Organizational behavior and human decision processes.

[8]  M. Sujan,et al.  Consumer Knowledge: Effects on Evaluation Strategies Mediating Consumer Judgments , 1985 .

[9]  A. Comrey.  A proposed method for absolute ratio scaling , 1950, Psychometrika.

[10]  John W. Tukey,et al.  Data Analysis and Regression: A Second Course in Statistics , 1977 .

[11]  Orfelio G. León.  On the Death of SMART and the Birth of GRAPA , 1997 .

[12]  James Jaccard,et al.  A comparison of six methods for assessing the importance of perceived consequences in behavioral decisions: Applications from attitude research , 1984 .

[13]  F. H. Barron,et al.  SMARTS and SMARTER: Improved Simple Methods for Multiattribute Utility Measurement , 1994 .

[14]  Martin Weber,et al.  The Effect of Attribute Ranges on Weights in Multiattribute Utility Measurements , 1993 .

[15]  D. Brinberg,et al.  Assessing attribute importance: a comparison of six methods , 1986 .

[16]  Rodney H. Green,et al.  Testing the Reliability of Weight Elicitation Methods: Direct Rating versus Point Allocation , 2000 .

[17]  J. W. Hutchinson,et al.  Dimensions of Consumer Expertise , 1987 .

[18]  C. C. Waid,et al.  An Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models , 1982 .

[19]  Raimo P. Hämäläinen,et al.  On the convergence of multiattribute weighting methods , 2001, Eur. J. Oper. Res..

[20]  Roger M. Heeler,et al.  Attribute Importance: Contrasting Measurements , 1979 .

[21]  D. Winterfeldt,et al.  Comparison of weighting judgments in multiattribute utility measurement , 1991 .

[22]  G. W. Fischer.  Range Sensitivity of Attribute Weights in Multiattribute Value Models , 1995 .

[23]  Martin Weber,et al.  Behavioral influences on weight judgments in multiattribute decision making , 1993 .