The agreement chart

BackgroundWhen assessing the concordance between two methods of measurement of ordinal categorical data, summary measures such as Cohen’s (1960) kappa or Bangdiwala’s (1985) B-statistic are used. However, a picture conveys more information than a single summary measure.MethodsWe describe how to construct and interpret Bangdiwala’s (1985) agreement chart and illustrate its use in visually assessing concordance in several example clinical applications.ResultsThe agreement charts provide a visual impression that no summary statistic can convey, and summary statistics reduce the information to a single characteristic of the data. However, the visual impression is personal and subjective, and not usually reproducible from one reader to another.ConclusionsThe agreement chart should be used to complement the summary kappa or B-statistics, not to replace them. The graphs can be very helpful to researchers as an early step to understand relationships in their data when assessing concordance.

[1]  L. Kurland,et al.  Studies on multiple sclerosis in Winnipeg, Manitoba, and New Orleans, Louisiana. II. A controlled investigation of factors in the life history of the Winnipeg patients. , 1953, American journal of hygiene.

[2]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[3]  Stephen E. Fienberg,et al.  Discrete Multivariate Analysis: Theory and Practice , 1976 .

[4]  Annette J. Dobson,et al.  General observer-agreement measures on individual subjects and groups of subjects , 1984 .

[5]  V. Shankar,et al.  Behavior of agreement measures in the presence of zero cells and biased marginal distributions , 2008 .

[6]  Nitin R. Patel,et al.  A Network Algorithm for Performing Fisher's Exact Test in r × c Contingency Tables , 1983 .

[7]  B. Tung,et al.  Nosological coding of cause of death. , 1983, American journal of epidemiology.

[8]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[9]  Chandler Stolp,et al.  The Visual Display of Quantitative Information , 1983 .

[10]  J. Fleiss,et al.  Measuring Agreement for Multinomial Data , 1982 .

[11]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[12]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[13]  Beat Kleiner,et al.  A Mosaic of Television Ratings , 1984 .

[14]  N. L. Johnson,et al.  Linear Statistical Inference and Its Applications , 1966 .

[15]  Dana Quade,et al.  The pair chart , 1973 .

[16]  J. Shields.  Finger Prints , 1967 .

[17]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .

[18]  J. R. Landis,et al.  An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. , 1977, Biometrics.

[19]  S. Haberman,et al.  The analysis of frequency data , 1974 .

[20]  R. Gnanadesikan,et al.  Probability plotting methods for the analysis of data. , 1968, Biometrika.

[21]  J. A. Hartigan,et al.  Mosaics for Contingency Tables , 1981 .

[22]  R. Light.  Measures of response agreement for qualitative data: Some generalizations and alternatives. , 1971 .

[23]  Taylor Francis Online,et al.  The American statistician , 1947 .

[24]  L. Kurland,et al.  Studies on multiple sclerosis in Winnepeg, Manitoba, and New Orleans, Louisiana. I. Prevalence; comparison between the patient groups in Winnipeg and New Orleans. , 1953, American journal of hygiene.

[25]  C. Ireland,et al.  Analysis of frequency data. , 2010 .

[26]  H. Kraemer,et al.  Extension of the kappa coefficient. , 1980, Biometrics.

[27]  Q. Mcnemar.  Note on the sampling error of the difference between correlated proportions or percentages , 1947, Psychometrika.

[28]  J. R. Landis,et al.  A one-way components of variance model for categorical data , 1977 .

[29]  B. Everitt,et al.  Statistical methods for rates and proportions , 1973 .

[30]  Jacob Cohen,et al.  The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability , 1973 .

[31]  S. Bangdiwala,et al.  Interpretation of Kappa and B statistics measures of agreement , 1997 .

[32]  R. Prineas,et al.  Comparisons of cause of death verification methods and costs in the lipid research clinics program mortality follow-up study. , 1985, Controlled clinical trials.

[33]  M. Kendall,et al.  The Problem of $m$ Rankings , 1939 .

[34]  Sheila K West,et al.  Field evaluation of a rapid point-of-care assay for targeting antibiotic treatment for trachoma control: a comparative study , 2006, The Lancet.