Variation of preference inconsistency when applying ratio and interval scale pairwise comparisons

Several studies on numerical rating in discrete choice problems address the tendency of inconsistencies in decision makers' measured preferences. This is partly due to true inconsistencies in preferences or the decision makers' uncertainty on what he or she really wants. This uncertainty may be reflected in the elicited preferences in different ways depending on the questions asked and methods used in deriving the preferences for alternatives. Some part of the inconsistency is due to only having a discrete set of possible judgments. This study examined the variation of preference inconsistency when applying different pairwise preference elicitation techniques in a five-item discrete choice problem. The study data comprised preferences of five career alternatives elicited applying interval scale and numerically and verbally anchored ratio scale pairwise comparisons. Statistical regression technique was used to analyse the differences of inconsistencies between the tested methods. The resulting relative residual variances showed that the interval ratio scale comparison technique provided the greatest variation of inconsistencies between respondents, thus being the most sensitive to inconsistency in preferences. The numeric ratio scale comparison gave the most uniform preferences between the respondents. The verbal ratio scale comparison performed between the latter two when relative residual variances were considered. However, the verbal ratio scale comparison had weaker ability to differentiate the alternatives. The results indicated that the decision recommendation may not be sensitive to the selection between these preference elicitation methods in this kind of five-item discrete choice problem. The numeric ratio scale comparison technique seemed to be the most suitable method to reveal the decision makers' true preferences. However, to confirm this result, more studying will be needed, with an attention paid to users' comprehension and learning in the course of the experiment. Copyright © 2013 John Wiley & Sons, Ltd.

[1]  Bernard Roy,et al.  Classement et choix en présence de points de vue multiples , 1968 .

[2]  P. D. Jong.  A statistical approach to Saaty's scaling method for priorities , 1984 .

[3]  G. Crawford,et al.  A note on the analysis of subjective judgment matrices , 1985 .

[4]  Jean Pierre Brans,et al.  HOW TO SELECT AND HOW TO RANK PROJECTS: THE PROMETHEE METHOD , 1986 .

[5]  F. Lootsma.  SCALE SENSITIVITY IN THE MULTIPLICATIVE AHP AND SMART , 1993 .

[6]  Jyrki Kangas,et al.  Uncertainty in Expert Predictions of the Ecological Consequences of Forest Plans , 1996 .

[7]  Waldemar W. Koczkodaj,et al.  A Monte Carlo Study of Parwise Comparison , 1996, Inf. Process. Lett..

[8]  Jyrki Kangas,et al.  Analyzing uncertainties in experts' opinions of forest plan performance , 1997 .

[9]  R. Hämäläinen,et al.  On the measurement of preferences in the analytic hierarchy process , 1997 .

[10]  Martin Rogers,et al.  A new system for weighting environmental criteria for use within ELECTRE III , 1998, Eur. J. Oper. Res..

[11]  Jyrki Kangas,et al.  Analysing uncertainties of interval judgment data in multiple-criteria evaluation of forest plans , 1998 .

[12]  John W. Payne,et al.  Measuring Constructed Preferences: Towards a Building Code , 1999 .

[13]  P. Leskinen.  Measurement scales and scale independence in the Analytic Hierarchy Process , 2000 .

[14]  Annika Kangas,et al.  HERO: Heuristic Optimisation for Multi-Criteria Forestry Decision Analysis , 2001 .

[15]  John R. Doyle,et al.  A comparison of three weight elicitation methods: good, better, and best , 2001 .

[16]  P. Leskinen,et al.  Regression Methods for Pairwise Comparison Data , 2001 .

[17]  T. Saaty.  The Seven Pillars of the Analytic Hierarchy Process , 2001 .

[18]  T. Saaty.  Fundamentals of the Analytic Hierarchy Process , 2001 .

[19]  David L. Olson,et al.  Ordinal judgments in multiattribute decision analysis , 2002, Eur. J. Oper. Res..

[20]  Annika Kangas,et al.  Multiple Criteria Decision Support Methods in Forest Management , 2002 .

[21]  J. Kangas,et al.  Assessing ecological values with dependent explanatory variables in multi-criteria forest ecosystem management , 2003 .

[22]  Miroslaw Kwiesielewicz,et al.  Inconsistent and contradictory judgements in pairwise comparison method in the AHP , 2004, Comput. Oper. Res..

[23]  H. Vacik,et al.  Application of the analytic network process in multi-criteria analysis of sustainable forest management , 2005 .

[24]  Jonathan Barzilai,et al.  Measurement and preference function modelling , 2005, Int. Trans. Oper. Res..

[25]  Jeffrey E. Kottemann,et al.  User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques , 2006, Eur. J. Oper. Res..

[26]  Bernard Yannou,et al.  Selective assessment of judgmental inconsistencies in pairwise comparisons for group decision rating , 2007, Comput. Oper. Res..

[27]  Carlos A. Bana e Costa,et al.  A critical analysis of the eigenvalue method used to derive priorities in AHP , 2008, Eur. J. Oper. Res..

[28]  Tamás Rapcsák,et al.  On Saaty’s and Koczkodaj’s inconsistencies of pairwise comparison matrices , 2008, J. Glob. Optim..

[29]  Pedro Linares,et al.  Are inconsistent decisions better? An experiment with pairwise comparisons , 2009, Eur. J. Oper. Res..

[30]  T Kainulainen,et al.  A statistical approach to assessing interval scale preferences in discrete choice problems , 2009, J. Oper. Res. Soc..

[31]  József Temesi,et al.  Pairwise comparison matrices and the error-free property of the decision maker , 2011, Central Eur. J. Oper. Res..