Why Generalizability Theory Yields Better Results than Classical Test Theory.

Generalizability theory provides a technique for accurately estimating the reliability of measurements. The power of this theory is based on the simultaneous analysis of multiple sources of error variances. Equally important, generalizability theory considers relationships among the sources of measurement error. Just as multivariate inferential statistics consider relationships among variables that univariate statistics cannot detect, generalizability theory considers relationships of error measurement that classical theory cannot. An extensive discussion of the concept of reliability and its use in classical test theory and generalizability theory is presented. A comparison of classical teat theory and generalizability theory illustrates how generalizability theory subsumes all other reliability estimates as special cases. A hypothetical data set provides examples of when the failure to use generalizability theory can lead to seriously erroneous estimates of test reliability. The framework of generalizability theory incorporates two stages of analysis: (1) a generalizability study; and (2) a decision study. The former analyzes the extent to which results are generalizable to a population, while ,he latter uses information from the generalizability study to determine other generalizability coefficients for variations of the measurement protocol. Six data tablet are provided, and an appendix presents the GENOVA program cede used. (TJH) ********* ***** ** * **** ** * ** * ** * **** ** * **** * ******** * * * ***** ** ** * ***** * ** Reproductions supplied by EDRS are the best that can be made from the original document. * **********************************************.************************ U S DEPARTMENT OF EDUCATION Office of Educational Resew, h and Improvement EDUCATIONAL RESOURCES .NcoP"ariON CENTER iERIC, 14hiS ()acument haS been rerrOduced as received from the person or organization originating it Minor changes have been made to improve reproductiOn duality Points of view or opinions stated in this docu ment do not necessarily represent nIttrial OE RI position o policy "PERMISSION TO REPRODUCE THIS MATERIAL IN MICROFICHE ONLY HAS BEEN GRANTED BY 5/9 tuz . i '9 6-456A) TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)" WHY GENERALIZABILITY THEORY YIELDS BETTER RESULTS THAN CLASSICAL TEST THEORY