Statistical Inference When Classroom Quality is Measured With Error

Abstract A dramatic shift in research priorities has recently produced a large number of ambitious randomized trials in K-12 education. In most cases, the aim is to improve student academic learning by improving classroom instruction. Embedded in these studies are theories about how the quality of classroom must improve if these interventions are to succeed. The problem of measuring classroom quality then emerges as a major concern. This article first considers how errors of measurement reduce statistical power in studies of the impact of interventions classroom quality. We show how to use information about reliability to compute power and plan new research. At the same time, errors of measurement introduce bias into estimates of the association between classroom quality and student outcomes. We show how to use knowledge about the magnitude of measurement error to eliminate or reduce this bias. We also briefly review research on the design of studies of the reliability of classroom measures. Such studies are essential to evaluate promising new classroom interventions.

[1]  Jere Brophy,et al.  Teachers' communication of differential expectations for children's classroom performance: Some beha , 1970 .

[2]  Stephen W. Raudenbush,et al.  1. Ecometrics: Toward a Science of Assessing Ecological Settings, with Application to the Systematic Social Observation of Neighborhoods , 1999 .

[3]  L. Cronbach,et al.  Psychological tests and personnel decisions , 1958 .

[4]  Eric M. Camburn,et al.  Using Teacher Logs to Measure the Enacted Curriculum: A Study of Literacy Teaching in Third‐Grade Classrooms , 2004, The Elementary School Journal.

[5]  Andrew J. Mashburn,et al.  Measures of classroom quality in prekindergarten and children's development of academic, language, and social skills. , 2008, Child development.

[6]  Laura R. Peck Subgroup Analysis in Social Experiments: Measuring Program Impacts Based on Post-Treatment Choice , 2003 .

[7]  P. Rossi,et al.  Evaluation: A systematic approach, 5th ed. , 1989 .

[8]  C. Achilles,et al.  Evaluation: A Systematic Approach , 1980 .

[9]  J. Gill Hierarchical Linear Models , 2005 .

[10]  Lee J. Cronbach,et al.  Psychological tests and personnel decisions , 1958 .

[11]  S. Raudenbush,et al.  Statistical power and optimal design for multisite randomized trials. , 2000, Psychological methods.

[12]  H. Goldstein Multilevel Statistical Models , 2006 .

[13]  R. Gallimore,et al.  Using Video Surveys to Compare Classrooms and Teaching Across Cultures: Examples and Lessons From the TIMSS Video Studies , 2000 .

[14]  Risto Lethonen Multilevel Statistical Models (3rd ed.) , 2005 .

[15]  S. Raudenbush Statistical analysis and optimal design for cluster randomized trials , 1997 .