Models for Paired Comparison Data: A Review with Emphasis on Dependent Data

Thurstonian and Bradley-Terry models are the most commonly applied models in the analysis of paired comparison data. Since their introduction, numerous developments have been proposed in different areas. This paper provides an updated overview of these extensions, including how to account for object- and subject-specific covariates and how to deal with ordinal paired comparison data. Special emphasis is given to models for dependent comparisons. Although these models are more realistic, their use is complicated by numerical difficulties. We therefore concentrate on implementation issues. In particular, a pairwise likelihood approach is explored for models for dependent paired comparison data, and a simulation study is carried out to compare the performance of maximum pairwise likelihood with other limited information estimation methods. The methodology is illustrated throughout using a real data set about university paired comparisons performed by students.

[1]  U Böckenholt,et al.  Individual differences in paired comparison data. , 2001, The British journal of mathematical and statistical psychology.

[2]  B. Francis,et al.  Modeling heterogeneity in ranked responses by nonparametric maximum likelihood: How do Europeans get their scientific knowledge? , 2010, 1101.1425.

[3]  R. Tsai,et al.  Remarks on the identifiability of thurstonian ranking models: Case V, case III, or neither? , 2000 .

[4]  Rainer Schwabe,et al.  Optimal design for the Bradley–Terry paired comparison model , 2008, Stat. Methods Appl..

[5]  Ulf Böckenholt,et al.  Structural equation modeling of paired-comparison and ranking data. , 2005, Psychological methods.

[6]  Harry Joe,et al.  A General Family of Limited Information Goodness-of-Fit Statistics for Multinomial Data , 2010 .

[7]  Sylvain Choisel,et al.  Evaluation of multichannel reproduced sound: scaling auditory attributes underlying listener preference. , 2007, The Journal of the Acoustical Society of America.

[8]  Ulf Böckenholt,et al.  On the importance of distinguishing between within- and between-subject effects in intransitive intertemporal choice , 2008 .

[9]  K. Bäuml,et al.  Effects of the beholder's age on the perception of facial attractiveness. , 2000, Acta psychologica.

[10]  David Firth,et al.  Bradley-Terry Models in R: The BradleyTerry2 Package , 2012 .

[11]  David Causeur,et al.  A 2-dimensional extension of the Bradley–Terry model for paired comparisons , 2005 .

[12]  Ulf Böckenholt,et al.  Comparative judgments as an alternative to ratings: identifying the scale origin. , 2004, Psychological methods.

[13]  D. Culler,et al.  Comparison of methods , 2000 .

[14]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[15]  Regina Dittrich,et al.  Modelling dependencies in paired comparison data a log-linear approach , 2002 .

[16]  Peter Goos,et al.  Optimal design of factorial paired comparison experiments in the presence of within-pair order effects , 2011 .

[17]  David Firth,et al.  Generalized nonlinear models in R: An overview of the gnm package , 2007 .

[18]  William R. Dillon,et al.  Capturing Individual Differences in Paired Comparisons: An Extended BTL Model Incorporating Descriptor Variables , 1993 .

[19]  L. V. Jones,et al.  The Rational Origin for Measuring Subjective Values , 1957 .

[20]  Ulf Böckenholt,et al.  A logistic representation of multivariate paired-comparison models , 1988 .

[21]  H. Joe,et al.  Composite likelihood estimation in multivariate data analysis , 2005 .

[22]  Suzanne Winsberg,et al.  A thurstonian pairwise choice model with univariate and multivariate spline transformations , 1993 .

[23]  J. C. van Houwelingen,et al.  Logistic Regression for Correlated Binary Data , 1994 .

[24]  Yoshio Takane,et al.  Analysis of Covariance Structures and Probabilistic Binary Choice Data , 1989 .

[25]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[26]  E. Zermelo Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung , 1929 .

[27]  Ulf Böckenholt,et al.  A Thurstonian analysis of preference change , 2002 .

[28]  Daniel Barry,et al.  Choice models for predicting divisional winners in major league baseball , 1993 .

[29]  H. A. David,et al.  The Method of Paired Comparisons (2nd ed.). , 1989 .

[30]  D. Hunter MM algorithms for generalized Bradley-Terry models , 2003 .

[31]  Achim Zeileis,et al.  Accounting for Individual Differences in Bradley-Terry Models by Means of Recursive Partitioning , 2011 .

[32]  Frank Bretz,et al.  Comparison of Methods for the Computation of Multivariate t Probabilities , 2002 .

[33]  Ulf Böckenholt,et al.  14 Random-Effects Models for Preference Data , 2006 .

[34]  L. Knorr‐Held Dynamic Rating of Sports Teams , 2000 .

[35]  Joan L. Walker,et al.  Generalized random utility model , 2002, Math. Soc. Sci..

[36]  B. Muthén Contributions to factor analysis of dichotomous variables , 1978 .

[37]  S. Chib,et al.  Analysis of multivariate probit models , 1998 .

[38]  David Mease,et al.  A Penalized Maximum Likelihood Approach for the Ranking of College Football Teams Independent of Victory Margins , 2003 .

[39]  Chih-Jen Lin,et al.  Generalized Bradley-Terry Models and Multi-Class Probability Estimates , 2006, J. Mach. Learn. Res..

[40]  R. Henery,et al.  An extension to the Thurstone-Mosteller model for chess , 1992 .

[41]  D. Quade,et al.  Random effects in paired-comparison experiments using the Bradley-Terry model. , 1983, Biometrics.

[42]  P. V. Rao,et al.  Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model , 1967 .

[43]  C. Varin,et al.  Dynamic Bradley–Terry modelling of sports tournaments , 2013 .

[44]  Ulf Böckenholt,et al.  Thurstonian-Based Analyses: Past, Present, and Future Utilities , 2006, Psychometrika.

[45]  Regina Dittrich,et al.  Analysing partial ranks by using smoothed paired comparison methods: an investigation of value orientation in Europe , 2002 .

[46]  D. Curtis,et al.  An extended transmission/disequilibrium test (TDT) for multi‐allele marker loci , 1995, Annals of human genetics.

[47]  B. Muthén,et al.  Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes , 1997 .

[48]  Albert Maydeu-Olivares,et al.  Limited information estimation and testing of Thurstonian models for paired comparison data under multiple judgment sampling , 2001 .

[49]  R. Davidson On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments , 1970 .

[50]  J. Kent Robust properties of likelihood ratio tests , 1982 .

[51]  David Firth,et al.  Ultraviolet signals ultra-aggression in a lizard , 2006, Animal Behaviour.

[52]  R. Duncan Luce,et al.  Individual Choice Behavior: A Theoretical Analysis , 1979 .

[53]  H. Joe,et al.  Limited-and Full-Information Estimation and Goodness-ofFit Testing in 2 n Contingency Tables : A Unified Framework , 2005 .

[54]  D. Firth Bias reduction of maximum likelihood estimates , 1993 .

[55]  Harry Joe,et al.  Limited Information Goodness-of-Fit Testing in Multidimensional Contingency Tables , 2005 .

[56]  C. Sinclair,et al.  GLIM for Preference , 1982 .

[57]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[58]  Harry Joe,et al.  Extended Use of Paired Comparison Models, with Application to Chess Rankings , 1990 .

[59]  L. R. Ford Solution of a Ranking Problem from Binary Comparisons , 1957 .

[60]  Yi-Ching Yao,et al.  Asymptotics when the number of parameters tends to infinity in the Bradley-Terry model for paired comparisons , 1999 .

[61]  S. Stigler Citation Patterns in the Journals of Statistics and Probability , 1994 .

[62]  U Böckenholt Hierarchical modeling of paired comparison data. , 2001, Psychological methods.

[63]  D. Cox,et al.  A note on pseudolikelihood constructed from marginal densities , 2004 .

[64]  Albert Maydeu-Olivares,et al.  Limited Information Goodness-of-fit Testing in Multidimensional Contingency Tables , 2005 .

[65]  Christian Schmid,et al.  A Matlab function to estimate choice model parameters from paired-comparison data , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[66]  Albert Maydeu-Olivares,et al.  Limited Information Estimation and Testing of Discretized Multivariate Normal Structural Models , 2003 .

[67]  F. Mosteller,et al.  Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed , 1951, Psychometrika.

[68]  David Firth,et al.  Multiple signals in chameleon contests: designing and analysing animal contests as a tournament , 2006, Animal Behaviour.

[69]  Ulf Böckenholt,et al.  Modeling within-subject dependencies in ordinal paired comparison data , 1997 .

[70]  Tony R. Martinez,et al.  A Bradley–Terry artificial neural network model for individual ratings in group competitions , 2008, Neural Computing and Applications.

[71]  Alberto Maydeu-Olivares,et al.  Identification and Small Sample Estimation of Thurstone's Unrestricted Model for Paired Comparisons Data , 2007, Multivariate behavioral research.

[72]  Ulf Böckenholt,et al.  Thresholds and Intransitivities in Pairwise Judgments: A Multilevel Analysis , 2001 .

[73]  Manuela Cattelan Correlation models for paired comparison data , 2009 .

[74]  A. Springall,et al.  Response Surface Fitting Using a Generalization of the Bradley‐Terry Paired Comparison Model , 1973 .

[75]  Wolfgang Ellermeier,et al.  Scaling the Unpleasantness of Sounds According to the BTL Model: Ratio-Scale Representation and Psychoacoustical Analysis , 2004 .

[76]  Heinz Holling,et al.  Optimal designs for main effects in linear paired comparison models , 2004 .

[77]  L. Thurstone A law of comparative judgment. , 1994 .

[78]  Anthony J. Hayter,et al.  The evaluation of general non‐centred orthant probabilities , 2003 .

[79]  Roger R. Davidson,et al.  A Bibliography on the Method of Paired Comparisons , 1973 .

[80]  H. A. David,et al.  Ties in Paired-Comparison Experiments Using a Modified Thurstone-Mosteller Model , 1960 .

[81]  G. Molenberghs,et al.  Models for Discrete Longitudinal Data , 2005 .

[82]  Ulf Böckenholt,et al.  Modelling intransitive preferences: A random-effects approach , 2006 .

[83]  Brian Francis,et al.  Modelling dependency in multivariate paired comparisons: A log-linear approach , 2006, Math. Soc. Sci..

[84]  Alan Agresti,et al.  Analysis of Ordinal Paired Comparison Data , 1992 .

[85]  K. Bäuml,et al.  Upright versus upside-down faces: How interface attractiveness varies with orientation , 1994, Perception & psychophysics.

[86]  Ian G. McHale,et al.  A Bradley-Terry type model for forecasting tennis match results , 2011 .

[87]  K. Train Discrete Choice Methods with Simulation , 2003 .

[88]  Ulf Böckenholt,et al.  Two-level linear paired comparison models: estimation and identifiability issues , 2002, Math. Soc. Sci..

[89]  Hal S. Stern,et al.  A continuum of paired comparisons models , 1990 .

[90]  N. Breslow,et al.  Approximate inference in generalized linear mixed models , 1993 .

[91]  Thomas A. Mazzuchi,et al.  A paired comparison experiment for gathering expert judgment for an aircraft wiring risk assessment , 2008, Reliab. Eng. Syst. Saf..

[92]  Walter Katzenbeisser,et al.  A log-linear approach for modelling ordinal paired comparison data on motives to start a PhD programme , 2004 .

[93]  Megan L. Head,et al.  Chemical mediation of reciprocal mother-offspring recognition in the Southern Water Skink (Eulamprus heatwolei) , 2008 .

[94]  C.A.A. Duineveld,et al.  Log-linear modelling of paired comparison data from consumer tests , 2000 .

[95]  Ulf Böckenholt,et al.  Modeling subjective health outcomes: top 10 reasons to use Thurstone's method. , 2008, Medical care.

[96]  R. Tsai,et al.  Remarks on the identifiability of thurstonian paired comparison models under multiple judgment , 2003 .

[97]  Ulf Böckenholt,et al.  Some New Methods for an Old Problem: Modeling Preference Changes and Competitive Market Structures in Pretest Market Data: , 1997 .

[98]  Albert Maydeu-Olivares Thurstonian covariance and correlation structures for multiple judgment paired comparison data , 2001 .

[99]  Peter S. Craig,et al.  A new reconstruction of multivariate normal orthant probabilities , 2007 .

[100]  Steven Stern,et al.  Moderated paired comparisons: a generalized Bradley–Terry model for continuous data using a discontinuous penalized likelihood function , 2011 .

[101]  A. Tversky Elimination by aspects: A theory of choice. , 1972 .

[102]  J. Marschak Binary Choice Constraints on Random Utility Indicators , 1959 .

[103]  L. Fahrmeir,et al.  Dynamic Stochastic Models for Time-Dependent Ordered Paired Comparison Systems , 1994 .

[104]  Satoshi Usami,et al.  INDIVIDUAL DIFFERENCES MULTIDIMENSIONAL BRADLEY-TERRY MODEL USING REVERSIBLE JUMP MARKOV CHAIN MONTE CARLO ALGORITHM , 2010 .

[105]  Mark E. Glickman,et al.  Dynamic paired comparison models with stochastic variances , 2001 .

[106]  N. Reid,et al.  AN OVERVIEW OF COMPOSITE LIKELIHOOD METHODS , 2011 .

[107]  D. Firth Bradley-Terry Models in R , 2005 .

[108]  Albert Maydeu-Olivares,et al.  Limited information estimation and testing of Thurstonian models for preference data , 2002, Math. Soc. Sci..

[109]  Mark Reiser,et al.  Goodness-of-fit testing using components based on marginal frequencies of multinomial data. , 2008, The British journal of mathematical and statistical psychology.

[110]  Carolyn Hicks,et al.  Designing sexual health services for young people: a methodology for capturing the user voice. , 2009, Health & social care in the community.

[111]  R A Bradley Science, statistics, and paired comparisons. , 1976, Biometrics.

[112]  J. Douglas Carroll,et al.  Toward a new paradigm for the study of multiattribute choice behavior: Spatial and discrete modeling of pairwise preferences. , 1991 .

[113]  Regina Dittrich,et al.  Missing observations in paired comparison data , 2012 .

[114]  Reinhold Hatzinger,et al.  Fitting paired comparison models in R. , 2004 .

[115]  Harry Joe,et al.  Composite Likelihood Methods , 2012 .

[116]  Jinfeng Xu,et al.  GROUPED SPARSE PAIRED COMPARISONS IN THE BRADLEY-TERRY MODEL , 2011, 1111.5110.

[117]  Subhash R. Lele,et al.  Estimability and Likelihood Inference for Generalized Linear Mixed Models Using Data Cloning , 2010 .

[118]  J. Matthews,et al.  An application of Bradley-Terry-type models to the measurement of pain , 1995 .

[119]  Regina Dittrich,et al.  A paired comparison approach for the analysis of sets of Likert-scale responses , 2007 .

[120]  Regina Dittrich,et al.  Modelling the effect of subject‐specific covariates in paired comparison studies with an application to university rankings , 2001 .

[121]  Brian Francis,et al.  WU Institutional Repository , 2018 .