Consistency and Accuracy in Decision Aids: Experiments with Four Multiattribute Systems*

There have been a number of multiattribute decision aids developed to aid selection problems. Multiattribute value theory and the analytic hierarchy process are two commonly used techniques. Different systems can result in radically different conclusions if they inaccurately and inconsistently reflect the preference structure of decision makers, or if they are based on inappropriate theoretical models. This study examines the impact of the underlying theoretical model, the method in which preference information is elicited, and the structure of alternatives as influences on the results from using various decision aids. It was found that two systems based on the multiattribute value theory model were just as diverse in their conclusions as were results between AHP and the multiattribute value theory models. Therefore, accuracy of information reflecting decision maker preference is an important consideration. Feedback capable of assuring the decision maker that information provided is consistent is a necessary feature required of decision aids applied to selection problems. The study also found that the way in which information is elicited influenced the result more than did the underlying model. Exact numerical data for complex concepts such as attribute importance and alternative performance on attributes is not necessary, and elicitation procedures that are more natural for the user are likely to be more accurate.

[1]  Jung P. Shim,et al.  Bibliographical research on the Analytic Hierarchy Process (AHP) , 1989 .

[2]  Peter H. Farquhar,et al.  OR Forum - Perspectives on Utility Theory , 1986, Oper. Res..

[3]  James L. Corner,et al.  Decision Analysis Applications in the Operations Research Literature, 1970-1989 , 1991, Oper. Res..

[4]  Larry D. Rosen,et al.  An eye fixation analysis of multialternative choice , 1975, Memory & cognition.

[5]  P. Humphreys,et al.  Experiences with MAUD: Aiding decision structuring versus bootstrapping the decision maker☆ , 1980 .

[6]  P. Pardalos,et al.  Minimization approach to membership evaluation in fuzzy sets and error analysis , 1990 .

[7]  David L. Olson,et al.  Review of Empirical Studies in Multiobjective Mathematical Programming: Subject Reflection of Nonlinear Utility and Learning , 1992 .

[8]  Gregory W. Fischer,et al.  UTILITY MODELS FOR MULTIPLE OBJECTIVE DECISIONS: DO THEY ACCURATELY REPRESENT HUMAN PREFERENCES?* , 1979 .

[9]  Daniel J. Power,et al.  AN EMPIRICAL ASSESSMENT OF COMPUTER‐ASSISTED DECISION ANALYSIS* , 1986 .

[10]  R. Dawes,et al.  Linear models in decision making. , 1974 .

[11]  D. Gustafson,et al.  Decision support systems effectiveness: Conceptual framework and empirical evaluation , 1990 .

[12]  P. Vincke,et al.  Note-A Preference Ranking Organisation Method: The PROMETHEE Method for Multiple Criteria Decision-Making , 1985 .

[13]  Stan Schenkerman,et al.  Use and Abuse of Weights in Multiple Objective Decision Support Models , 1991 .

[14]  F. Lootsma.  SCALE SENSITIVITY IN THE MULTIPLICATIVE AHP AND SMART , 1993 .

[15]  D. Winterfeldt,et al.  Comparison of weighting judgments in multiattribute utility measurement , 1991 .

[16]  Valerie Belton,et al.  On a short-coming of Saaty's method of analytic hierarchies , 1983 .

[17]  Oleg I. Larichev,et al.  ZAPROS-LM — A method and system for ordering multiattribute alternatives , 1995 .

[18]  Evangelos Triantaphyllou,et al.  An examination of the effectiveness of multi-dimensional decision-making methods: A decision-making paradox , 1989, Decis. Support Syst..

[19]  Panos M. Pardalos,et al.  On the Evaluation and Application of Different Scales For Quantifying Pairwise Comparisons in Fuzzy Sets , 1994 .

[20]  C. C. Waid,et al.  An Experimental Comparison of Different Approaches to Determining Weights in Additive Utility Models , 1982 .

[21]  R. Hogarth,et al.  Unit weighting schemes for decision making , 1975 .

[22]  V. Belton.  A comparison of the analytic hierarchy process and a simple multi-attribute value function , 1986 .

[23]  Boaz Golany,et al.  Deriving weights from pairwise comparison matrices: The additive case , 1990 .

[24]  Charles Vlek,et al.  WHAT CONSTITUTES A GOOD DECISION , 1984 .

[25]  Dennis M. Buede.  Software review. Overview of the MCDA software market , 1992 .

[26]  O. Larichev,et al.  Experiments comparing qualitative approaches to rank ordering of multiattribute alternatives , 1993 .

[27]  Bernard Roy,et al.  Classement et choix en présence de points de vue multiples , 1968 .

[28]  D. Bouyssou,et al.  Comparison of two decision-aid models applied to a nuclear power plant siting example , 1986 .

[29]  J. Dyer.  Remarks on the analytic hierarchy process , 1990 .

[30]  R. L. Winkler.  Decision modeling and rational choice: AHP and utility theory , 1990 .

[31]  Fu Li,et al.  EXPERIMENTAL EVALUATION OF MULTIPLE CRITERIA DECISION MODELS FOR APPLICATION TO WATER RESOURCES PLANNING , 1992 .

[32]  O. Larichev.  Cognitive validity in design of decision‐aiding techniques , 1992 .

[33]  D. A. Seaver,et al.  A comparison of weight approximation techniques in multiattribute utility decision making , 1981 .