A careful look at the importance of criteria and weights

We investigate the connection between weights, scales, and the importance of criteria, when a linear value function is assumed to be a suitable representation of a decision maker’s preferences. Our considerations are based on a simple two-criteria experiment, where the participants were asked to indicate which of the criteria was more important, and to pairwise compare a number of alternatives. We use the participants’ pairwise choices to estimate the weights for the criteria in such a way that the linear value function explains the choices to the extent possible. More specifically, we study two research questions: (1) is it possible to find a general scaling principle that makes the rank order of the importance of criteria consistent with the rank order of the magnitudes of the weights, and (2) how good is a simple, direct method of asking the decision maker to “provide” weights for the criteria compared to our estimation procedure. Our results imply that there is reason to question two common beliefs, namely that the values of the weights would reflect the importance of criteria, and that people could reliably “provide” such weights without estimation.