Algebraic exact inference for rater agreement models

In recent years, a method for sampling from conditional distributions for categorical data has been presented by Diaconis and Sturmfels. Their algorithm is based on the algebraic theory of toric ideals which are used to create so called “Markov Bases”. The Diaconis-Sturmfels algorithm leads to a non-asymptotic Monte Carlo Markov Chain algorithm for exact inference on some classes of models, such as log-linear models. In this paper we apply the Diaconis-Sturmfels algorithm to a set of models arising from the rater agreement problem with special attention to the multi-rater case. The relevant Markov bases are explicitly computed and some results for simplify the computation are presented. An extended example on a real data set shows the wide applicability of this methodology.

[1]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[2]  Stephen E. Fienberg,et al.  Discrete Multivariate Analysis: Theory and Practice , 1976 .

[3]  Fabio Rapallo Algebraic Markov Bases and MCMC for Two‐Way Contingency Tables , 2003 .

[4]  H. Wynn,et al.  Algebraic Statistics: Computational Commutative Algebra in Statistics , 2000 .

[5]  J. Forster,et al.  Monte Carlo exact tests for square contingency tables , 1996 .

[6]  A Agresti,et al.  Modelling patterns of agreement and disagreement , 1992, Statistical methods in medical research.

[7]  J. Darroch,et al.  Category Distinguishability and Observer Agreement , 1986 .

[8]  Stephen E. Fienberg,et al.  The analysis of cross-classified categorical data , 1980 .

[9]  Martin A. Tanner,et al.  Modeling Agreement among Raters , 1985 .

[10]  Neil Klar,et al.  An Estimating Equations Approach for Modelling Kappa , 2000 .

[11]  A Agresti,et al.  Exact inference for categorical data: recent advances and continuing controversies , 2001, Statistics in medicine.

[12]  Roberto La Scala,et al.  Computing Toric Ideals , 1999, J. Symb. Comput..

[13]  S. Haberman,et al.  The analysis of frequency data , 1974 .

[14]  Martin Kreuzer,et al.  Computational Commutative Algebra 1 , 2000 .

[15]  G.,et al.  A review of statistical methods in the analysis of data arising from observer reliability studies (Part II)* , 2007 .

[16]  S. Chib,et al.  Understanding the Metropolis-Hastings Algorithm , 1995 .

[17]  James A. Hanley,et al.  Receiver Operating Characteristic (ROC) Curves , 2005 .

[18]  J. Fleiss Statistical methods for rates and proportions , 1974 .

[19]  P. Diaconis,et al.  Algebraic algorithms for sampling from conditional distributions , 1998 .

[20]  A. Agresti Categorical data analysis , 1993 .

[21]  J. Pitman,et al.  Hitting, Occupation, and Inverse Local Times of One-Dimensional Diffusions: Martingale and Excursion , 2003 .

[22]  A Donner,et al.  Testing the equality of two dependent kappa statistics. , 2000, Statistics in medicine.

[23]  A Donner,et al.  The statistical analysis of kappa statistics in multiple samples. , 1996, Journal of clinical epidemiology.

[24]  A. Dobra Markov bases for decomposable graphical models , 2003 .

[25]  W R Dillon,et al.  A Probabilistic Latent Class Model for Assessing Inter-Judge Reliability. , 1984, Multivariate behavioral research.

[26]  M. Becker Quasisymmetric Models for the Analysis of Square Contingency Tables , 1990 .