Inferential procedures for multifaceted coefficients of generalizability

A two-facet measurement model with broad application in the behavioral sciences is identified, and its coefficient of generalizability (CG) is examined. A normalizing transformation is proposed, and an asymptotic variance expression is derived. Three other multifaceted measurement models and CGs are identified, and variance expressions are presented. Next, an empirical investigation of the procedures follows, and it is shown that, in most cases, Type I error control in inferential applications is precise, and that the estimates are relatively efficient compared with the correlation coefficient. Implications for further research and for practice are noted. In an Appendix, four additional models, CGs, and variance expressions are presented.

[1]  L. A. Marascuilo Large-sample multiple comparisons. , 1966, Psychological bulletin.

[2]  Donald Ervin Knuth,et al.  The Art of Computer Programming, Volume II: Seminumerical Algorithms , 1970 .

[3]  H. Kraemer Extension of Feldt's approach to testing homogeneity of coefficients of reliability , 1981 .

[4]  Leonard S. Feldt,et al.  A test of the hypothesis that Cronbach's alpha reliability coefficient is the same for two tests administered to the same sample , 1980 .

[5]  Philip L. Smith Sampling Errors of Variance Components in Small Sample Multifacet Generalizability Studies , 1978 .

[6]  Thomas E. Whalen,et al.  A k-sample significance test for independent alpha coefficients , 1976 .

[7]  Michael T. Kane,et al.  The Generalizability of Student Ratings of Instruction: Estimation of the Teacher and Course Components. , 1978 .

[8]  F. E. Satterthwaite Synthesis of variance , 1941 .

[9]  J. Fleiss,et al.  Approximate interval estimation for a certain intraclass correlation coefficient , 1978 .

[10]  E. Paulson,et al.  An Approximate Normalization of the Analysis of Variance Distribution , 1942 .

[11]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[12]  Donald Ervin Knuth,et al.  The Art of Computer Programming , 1968 .

[13]  Noreen M. Webb,et al.  Generalizability theory : 1973-1980 , 1991 .

[14]  Satterthwaite Fe An approximate distribution of estimates of variance components. , 1946 .

[15]  Norman L. Johnson,et al.  Sampling Distributions of Variance Components II. Empirical Studies of Unbalanced Nested Designs , 1966 .

[16]  Leonard S. Feldt,et al.  A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests , 1969 .

[17]  L. S. Feldt The approximate sampling distribution of Kuder-Richardson reliability coefficient twenty , 1965, Psychometrika.

[18]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[19]  L. Cronbach,et al.  Generalizability of stratified-parallel tests , 1965, Psychometrika.

[20]  Albert Nußbaum,et al.  Multivariate Generalizability Theory in Educational Measurement: An Empirical Study , 1984 .

[21]  N. Webb,et al.  Generalizability of General Education Development Ratings of Jobs in the United States , 1981 .

[22]  Donald E. Knuth,et al.  The art of computer programming. Vol.2: Seminumerical algorithms , 1981 .

[23]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[24]  Joseph L. Fleiss,et al.  On the Distribution of a Linear Combination of Independent Chi Squares , 1971 .

[25]  G. Box Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems, II. Effects of Inequality of Variance and of Correlation Between Errors in the Two-Way Classification , 1954 .

[26]  R. Hare,et al.  Generalizability of a checklist for assessment of psychopathy. , 1983, Journal of consulting and clinical psychology.

[27]  H. Rouanet,et al.  COMPARISON BETWEEN TREATMENTS IN A REPEATED‐MEASUREMENT DESIGN: ANOVA AND MULTIVARIATE METHODS , 1970 .

[28]  David J. Woodruff,et al.  Tests for equality of several alpha coefficients when their sample estimates are dependent , 1986 .

[29]  D. Chalmers,et al.  The Reliability of Ratings of the Familiarity of Environmental Stimuli , 1985 .

[30]  L. Cronbach,et al.  THEORY OF GENERALIZABILITY: A LIBERALIZATION OF RELIABILITY THEORY† , 1963 .

[31]  F. J. Smith,et al.  Language, time, and person effects on attitude scale translations. , 1977 .

[32]  P. Abrami The Generalizability of Student Ratings of Instruction. , 1977 .