On a Model for Concordance between Judges

CONSIDER a number n of judges, each of whom ranks the same set of k objects according to some particular criterion. Assume that each judge ranks (stochastically) independently of the other judges so that we regard the situation as that of n rankings (Daniels, 1950; Kendall, 1970) or of n related samples (Conover, 1971, p. 246). We wish to say something about whether and to what degree the judges act concordantly (or homogeneously) with respect to a particular ranking which is not assumed known beforehand. More particularly we wish to estimate this underlying ranking and also consider a model for the proposed measure of concordance. In so doing we address ourselves to the problem of non-null modelling referred to in Kendall (1970, p. V). To place the suggested model in its proper setting it is appropriate to review briefly the relevant literature on modelling non-null distributions for rankings. These models may be, broadly speaking, divided into three classes: (I) parametric; (II) paired comparison; and (III) sampling. The parametric approach may be further subdivided into the categories: (Ia) multivariate; and (Ib) independent deviations. They may be described as follows: