Aggregation and decision making using ranked data

Nonparametric procedures are frequently used to rank order alternatives. Often, information from several data sets must be aggregated to derive an overall ranking. When using nonparametric procedures, Simpson-like paradoxes can occur in which the conclusion drawn from the aggregate ranked data set seems contradictory to the conclusions drawn from the individual data sets. Extending previous results found in the literature for the Kruskal-Wallis test, this paper presents a strict condition that ranked data must satisfy in order to avoid this type of inconsistency when using nonparametric pairwise procedures or Bhapkar's V procedure to extract an overall ranking. Aggregating ranked data poses further difficulties because there exist numerous ways to combine ranked data sets. This paper illustrates these difficulties and derives an upper bound for the number of possible ways that two ranked data sets can be combined.

[1]  Raymond T. Stefani,et al.  Survey of the major world sports rating systems , 1997 .

[2]  W. Kruskal,et al.  Use of Ranks in One-Criterion Variance Analysis , 1952 .

[3]  Ronald Fagin,et al.  Efficient similarity search and classification via rank aggregation , 2003, SIGMOD '03.

[4]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[5]  Jayant Kalagnanam,et al.  A Computational Study of the Kemeny Rule for Preference Aggregation , 2004, AAAI.

[6]  E. Antonsson,et al.  Arrow's Theorem and Engineering Design Decision Making , 1999 .

[7]  N. Schofield.  The Geometry of Voting , 1983 .

[8]  P.-C.-F. Daunou,et al.  Mémoire sur les élections au scrutin , 1803 .

[9]  Michel Truchon,et al.  Aggregation of Rankings in Figure Skating , 2004 .

[10]  Thomas L. Saaty,et al.  Multicriteria Decision Making: The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation , 1990 .

[11]  Stuart Pugh,et al.  Total Design: Integrated Methods for Successful Product Engineering , 1991 .

[12]  Nir Ailon,et al.  Aggregating inconsistent information: Ranking and clustering , 2008 .

[13]  F. Wilcoxon.  Individual Comparisons by Ranking Methods , 1945 .

[14]  K. Arrow,et al.  Handbook of Social Choice and Welfare , 2011 .

[15]  D. Saari,et al.  The Lack of Consistency for Statistical Decision Procedures , 1991 .

[16]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[17]  Nicolas de Condorcet.  Essai Sur L'Application de L'Analyse a la Probabilite Des Decisions Rendues a la Pluralite Des Voix , 2009 .

[18]  Donald G. Saari,et al.  Symmetry of Nonparametric Statistical Tests on Three Samples , 2007 .

[19]  V. P. Bhapkar.  A Nonparametric Test for the Problem of Several Samples , 1961 .

[20]  Thomas Jech.  The Ranking of Incomplete Tournaments: A Mathematician's Guide to Popular Sports , 1983 .

[21]  Deanna B. Haunsperger,et al.  Aggregated statistical rankings are arbitrary , 2003, Soc. Choice Welf..