Does the Precision and Stability of Value-Added Estimates of Teacher Performance Depend on the Types of Students They Serve?

This paper investigates how the precision and stability of a teacher's value-added estimate relates to the characteristics of the teacher's students. Using a large administrative data set and a variety of teacher value-added estimators, it finds that the stability over time of teacher value-added estimates can depend on the previous achievement level of a teacher's students. The differences are large in magnitude and statistically significant. The year-to-year stability level of teacher value-added estimates are typically 25% to more than 50% larger for teachers serving initially higher performing students compared to teachers with initially lower performing students. In addition, some differences are detected even when the number of student observations is artificially set to the same level and the data are pooled across two years to compute teacher value-added. Finally, the paper offers a policy simulation which demonstrates that teachers who face students with certain characteristics may be differentially likely to be the recipient of sanctions in a high stakes policy based on value-added estimates and more likely to see their estimates vary from year-to-year due to low stability.

[1]  Thomas A Louis,et al.  Jump down to Document , 2022 .

[2]  Raymond L. Pecheone,et al.  Gathering Feedback for Teaching Combining High-Quality Observations with Student Surveys and Achievement Gains , 2012 .

[3]  Dan Goldhaber,et al.  Assessing the “Rothstein Falsification Test”: Does It Really Show Teacher Value-Added Models Are Biased? , 2015 .

[4]  F. Lord.  Applications of Item Response Theory To Practical Testing Problems , 1980 .

[5]  Mark D. Reckase,et al.  Evaluating Specification Tests in the Context of Value-Added Estimation , 2014 .

[6]  Peter Z. Schochet,et al.  What Are Error Rates for Classifying Teacher and School Performance Using Value-Added Models? , 2013 .

[7]  Brian A. Jacob,et al.  Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education , 2008, Journal of Labor Economics.

[8]  Petra E. Todd,et al.  On the Specification and Estimation of the Production Function for Cognitive Achievement , 2003 .

[9]  Mark D. Reckase,et al.  An Evaluation of Empirical Bayes’s Estimation of Value-Added Teacher Performance Measures , 2015 .

[10]  Tim R. Sass,et al.  Value-Added Models and the Measurement of Teacher Productivity. CALDER Working Paper No. 54. , 2010 .

[11]  Jesse Rothstein,et al.  Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement , 2008 .

[12]  Mark D. Reckase,et al.  A Comparison of Student Growth Percentile and Value-Added Models of Teacher Performance , 2014 .

[13]  Michael Hansen,et al.  Is it Just a Bad Class? Assessing the Long‐Term Stability of Estimated Teacher Performance , 2012 .

[14]  Thomas J. Kane,et al.  Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation , 2008 .

[15]  Thomas J. Kane,et al.  The Promise and Pitfalls of Using Imprecise School Accountability Measures , 2002 .

[16]  Matthew Wiswall.  The Dynamics of Teacher Quality , 2013 .

[17]  Mark D. Reckase,et al.  Can Value-Added Measures of Teacher Performance Be Trusted? , 2012, Education Finance and Policy.

[18]  Jeffrey M. Wooldridge,et al.  Solutions Manual and Supplementary Materials for Econometric Analysis of Cross Section and Panel Data , 2003 .

[19]  Daniel F. McCaffrey,et al.  The Intertemporal Variability of Teacher Effect Estimates , 2009, Education Finance and Policy.

[20]  S. Loeb,et al.  Strategic Involuntary Teacher Transfers and Teacher Performance: Examining Equity and Efficiency , 2013 .

[21]  Morgaen L. Donaldson,et al.  The New Educational Accountability: Understanding the Landscape of Teacher Evaluation in the Post-NCLB Era , 2016, Education Finance and Policy.

[22]  Mariesa A. Herrmann,et al.  Shrinkage of Value-Added Estimates and Characteristics of Students with Hard-to-Predict Achievement Levels , 2016 .

[23]  Daniel F. McCaffrey,et al.  Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. , 2013 .

[24]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[25]  Daniel F. McCaffrey,et al.  Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects , 2014 .

[26]  Susan Johnson.  Will VAMS Reinforce the Walls of the Egg-Crate School? , 2015 .

[27]  E. Hanushek.  Conceptual and Empirical Issues in the Estimation of Educational Production Functions , 1979 .

[28]  M. Reckase.  Multidimensional Item Response Theory , 2009 .

[29]  Eric Isenberg,et al.  Methods for Accounting for Co-Teaching in Value-Added Models , 2012 .

[30]  D. Harris,et al.  Editors’ Introduction: The Use of Teacher Value-Added Measures in Schools , 2015 .

[31]  Cory Koedel,et al.  Re-Examining the Role of Teacher Quality in the Educational Production Function. Working Paper 2007-03. , 2007 .

[32]  Raj Chetty,et al.  The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood , 2011 .

[33]  Matthew G. Springer,et al.  Using Student Test Scores to Measure Teacher Performance , 2015 .

[34]  Paul Wright,et al.  Controlling for Student Background in Value-Added Assessment of Teachers , 2004 .

[35]  Raj Chetty,et al.  Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates , 2014 .

[36]  Dan Goldhaber,et al.  Does the Model Matter? Exploring the Relationship Between Different Student Achievement-Based Teacher Assessments , 2012 .