The application of multiple-criteria utility theory to the evaluation of information systems

The use of multiple-criteria theory, a means of measuring degree of satisfaction of predetermined evaluative criteria, quantifying their relative importance, identifying trade offs among them, and determining the best tradeoffs, is proposed for the evaluation of information systems. Use of the proposed method is demonstrated via the evaluation of a hypothetical selective dissemination of information (SDI) system. The illustrative evaluation involves (1) selection of two sets of typically antithetic criteria: efficiency (the inner workings of the system) and effectiveness (its impact on its users); (2) ranking and scoring each criterion in the two sets; (3) selection of alternative configurations of the system under study; (4) scoring the specimen and the alternative systems in terms of the rated efficiency and effectiveness criteria; and (5) identifying the system that scores the highest. The comparative scores for the systems are determined by Ui = [(W′) (Si′) + (W″) (S″i)], where S″i and S′i denote the efficiency and effectiveness scores of the ith alternative, and W′ and W′ are weighting constants that reflect the computed relative importance of efficiency and effectiveness.