Variability in Pretest-Posttest Correlation Coefficients by Student Achievement Level. Washington, DC: U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences

State assessments are increasingly used as outcome measures for education evaluations. The scaling of state assessments produces variability in measurement error, with the conditional standard error of measurement increasing as average student ability moves toward the tails of the achievement distribution. This report examines the variability in pretest-posttest correlation coefficients of state assessment data for samples of low-performing, average-performing, and proficient students to illustrate how sample characteristics (including the measurement error of observed scores) affect pretest-posttest correlation coefficients. As an application, this report highlights how statistical power can be attenuated when correlation coefficients vary according to sample characteristics. Population achievement data from four states and two large districts in both English/Language Arts and Mathematics for three recent years are examined. The results confirm that pretest-posttest correlation coefficients are smaller for samples of low performers, reducing statistical power for impact studies. We also find substantial variation across state assessments. These findings suggest that it may be useful to assess the pretest-posttest correlation coefficients of state assessments for an intervention’s target population during the planning phase of a study. NCEE 2011-4033 U.S. DEPARTMENT OF EDUCATION This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences (IES), under Contract ED-04-CO-0112/0006. Disclaimer The Institute of Education Sciences at the U.S. Department of Education contracted with Mathematica Policy Research to develop a report documenting the variability in the pretestposttest correlation coefficients estimated from samples of students who had taken state proficiency assessments and showing how attenuation in pretest-posttest correlation coefficients could affect statistical power in education experiments targeted at low-performing students. The views expressed in this report are those of the authors, and they do not necessarily represent the opinions and positions of the Institute of Education Sciences or the U.S. Department of Education. U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance Rebecca A. Maynard Commissioner September 2011 This report is in the public domain. Although permission to reprint this publication is not necessary, the citation should be the following: Cole, Russell, Joshua Haimson, Irma Perez-Johnson, and Henry May. “Variability in PretestPosttest Correlation Coefficients by Student Achievement Level.” NCEE Reference Report 2011-4033. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2011. This report is available on the IES website at http://ncee.ed.gov. Alternate Formats Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113.

[1]  Jiaxiu Yang,et al.  A Randomized Trial Study of a Preschool Literacy Curriculum: The Importance of Implementation , 2009 .

[2]  M. R. Novick,et al.  Statistical Theories of Mental Test Scores. , 1971 .

[3]  Philip C. Abrami,et al.  Computer-Assisted Tutoring in Success for All: Reading Outcomes for First Graders , 2008 .

[4]  C. Spearman.  The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[5]  J. Mckillip,et al.  Fundamentals of item response theory , 1993 .

[6]  Lucy J. Miller,et al.  Randomized experiments for planning and evaluation : a practical guide , 1998 .

[7]  Peter Z. Schochet.  The Late Pretest Problem in Randomized Control Trials of Education Interventions , 2008 .

[8]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[9]  Michael J. Kolen,et al.  Scaling, Norming, and Equating , 2011 .

[10]  J. Neher.  A problem of multiple comparisons , 2011 .

[11]  Andrew P. Jaciw,et al.  Using State or Study-Administered Achievement Tests in Impact Evaluations. , 2010 .

[12]  S. Vaughn,et al.  Enhancing Social Studies Vocabulary and Comprehension for Seventh-Grade English Language Learners: Findings From Two Experimental Studies , 2009 .

[13]  Sean P. Corcoran,et al.  Evaluation of Teachers Trained Through Different Routes to Certification , 2010 .

[14]  Jessaca Spybrook,et al.  An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences , 2009 .

[15]  Steven Glazerman Daniel P Mayer Paul T Decker.  Alternative Routes to Teaching: The Impacts of Teach For America on Student Achievement and Other Outcomes , 2006 .

[16]  Bruce Thompson,et al.  Psychometrics is Datametrics: the Test is not Reliable , 2000 .

[17]  R. Brennan,et al.  Estimators of Conditional Scale‐Score Standard Errors of Measurement: A Simulation Study , 2000 .

[18]  Howard S. Bloom,et al.  The Core Analytics of Randomized Experiments for Social Research. MDRC Working Papers on Research Methodology. , 2006 .

[19]  T. C. Oshima,et al.  Standardized Conditional SEM: A Case for Conditional Reliability , 2007 .

[20]  Prentice Starkey,et al.  Effects of a Pre-Kindergarten Mathematics Intervention: A Randomized Experiment , 2008 .

[21]  Irma Perez-Johnson,et al.  SECRETARY , 1943, Keywords of Identity, Race, and Human Mobility in Early Modern England.

[22]  J M Bos,et al.  Using Cluster Random Assignment to Measure Program Impacts , 1999, Evaluation review.

[23]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .