A Framework for Assessing Broad Sense Agreement Between Ordinal and Continuous Measurements

Conventional agreement studies have been confined to addressing the sense of reproducibility, and therefore are limited to assessing measurements on the same scale. In this work, we propose a new concept, called “broad sense agreement,” which extends the classical framework of agreement to evaluate the capability of interpreting a continuous measurement in an ordinal scale. We present a natural measure for broad sense agreement. Nonparametric estimation and inference procedures are developed for the proposed measure along with theoretical justifications. We also consider longitudinal settings which involve agreement assessments at multiple time points. Simulation studies have demonstrated good performance of the proposed method with small sample sizes. We illustrate our methods via an application to a mental health study.

[1]  K. Wells,et al.  A structured interview version of the Hamilton Depression Rating Scale: evidence of reliability and versatility of administration. , 1990, Journal of psychiatric research.

[2]  A K Manatunga,et al.  Paroxetine for the prevention of depression induced by high-dose interferon alfa. , 2001, The New England journal of medicine.

[3]  J. Fleiss Measuring nominal scale agreement among many raters. , 1971 .

[4]  L. Lin Assay Validation Using the Concordance Correlation Coefficient , 1992 .

[5]  Ying Guo,et al.  Measuring Agreement of Multivariate Discrete Survival Times Using a Modified Weighted Kappa Coefficient , 2009, Biometrics.

[6]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[7]  Ulf Olsson,et al.  A Measure of Agreement for Interval or Nominal Multivariate Observations , 2001 .

[8]  Vernon M Chinchilli,et al.  A Class of Repeated Measures Concordance Correlation Coefficients , 2007, Journal of biopharmaceutical statistics.

[9]  Jacob Cohen A Coefficient of Agreement for Nominal Scales , 1960 .

[10]  Ayman Baklizi,et al.  Nonparametric estimation of P(X , 2006 .

[11]  E Svensson,et al.  Concordance between ratings using different scales for the same variable. , 2000, Statistics in medicine.

[12]  Ying Guo,et al.  Nonparametric Estimation of the Concordance Correlation Coefficient under Univariate Censoring , 2007, Biometrics.

[13]  A. S. Hedayat,et al.  A Unified Approach for Assessing Agreement for Continuous and Categorical Data , 2007, Journal of biopharmaceutical statistics.

[14]  H. Hotelling New Light on the Correlation Coefficient and its Transforms , 1953 .

[15]  Robert H. Somers,et al.  A new asymmetric measure of association for ordinal variables. , 1962 .

[16]  R. Fisher 014: On the "Probable Error" of a Coefficient of Correlation Deduced from a Small Sample. , 1921 .

[17]  L. Lin,et al.  A concordance correlation coefficient to evaluate reproducibility. , 1989, Biometrics.

[18]  P. X. Song,et al.  Multivariate Dispersion Models Generated From Gaussian Copula , 2000 .

[19]  P. McCullagh Analysis of Ordinal Categorical Data , 1985 .

[20]  Jorge Quiroz,et al.  Assessment of Equivalence Using a Concordance Correlation Coefficient in a Repeated Measurements Design , 2005, Journal of biopharmaceutical statistics.

[21]  H. Kraemer,et al.  Extension of the kappa coefficient. , 1980, Biometrics.

[22]  Huiman X Barnhart,et al.  Weighted Least‐Squares Approach for Comparing Correlated Kappa , 2002, Biometrics.

[23]  Bernard De Baets,et al.  ROC analysis in ordinal regression learning , 2008, Pattern Recognit. Lett..

[24]  A K Manatunga,et al.  Modeling kappa for measuring dependent categorical agreement data. , 2000, Biostatistics.

[25]  J. K. Benedetti,et al.  Sampling Behavior of Tests for Correlation in Two-Way Contingency Tables , 1977 .

[26]  M. Kendall The treatment of ties in ranking problems. , 1945, Biometrika.