Creating Comparability among Reliability Coefficients: The Case of Cronbach Alpha and Cohen Kappa

Cronbach alpha and Cohen kappa were compared and found to differ along two major facets. A fourfold classification system based on these facets clarifies the double contrast and produces a common metric allowing direct comparability. A new estimator, coefficient beta, is introduced in the process and is presented as a complement to coefficient alpha in estimating the psychometric properties of test scores and ratings.

[1]  Jacob Cohen,et al.  Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. , 1968 .

[2]  M. J. Allen.  Introduction to Measurement Theory , 1979 .

[3]  Louis Guttman,et al.  A basis for analyzing test-retest reliability , 1945, Psychometrika.

[4]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[5]  K. McGraw,et al.  Forming inferences about some intraclass correlation coefficients. , 1996 .

[6]  E. J. O'Brien.  Global Self-Esteem Scales: Unidimensional or Multidimensional? , 1985 .

[7]  M. Appelbaum,et al.  Psychometric methods. , 1989, Annual review of psychology.

[8]  B. gray-Little,et al.  An Item Response Theory Analysis of the Rosenberg Self-Esteem Scale , 1997 .

[9]  R. Jackson.  RELIABILITY OF MENTAL TESTS , 1939 .

[10]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[11]  C. Hoyt.  Test reliability estimated by analysis of variance , 1941 .

[12]  B. Everitt,et al.  Large sample standard errors of kappa and weighted kappa. , 1969 .

[13]  D. J. Lee.  Society and the Adolescent Self-Image , 1969 .

[14]  Jacob Cohen,et al.  The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability , 1973 .

[15]  Edward E. Cureton,et al.  Reliability and Validity: Basic Assumptions and Experimental Designs , 1965 .

[16]  G Becker,et al.  How important is transient error in estimating reliability? Going beyond simulation studies. , 2000, Psychological methods.

[17]  P. Mielke,et al.  A Generalization of Cohen's Kappa Agreement Measure to Interval Measurement and Multiple Raters , 1988 .

[18]  R. L. Ebel,et al.  Estimation of the reliability of ratings , 1951 .

[19]  B. Byrne,et al.  On the Structure of Adolescent Self-Concept. , 1986 .

[20]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[21]  R. Rosenthal.  Estimating effective reliabilities in studies that employ judges' ratings , 1973 .

[22]  M. Rosenberg.  Society and the adolescent self-image , 1966 .

[23]  Gilbert Becker.  How important is transient error in estimating reliability? Going beyond simulation studies. , 2000 .

[24]  Xitao Fan,et al.  Published Studies of Interrater Reliability Often Overestimate Reliability: Computing the Correct Coefficient , 2000 .