Revisiting The Widget Effect: Teacher Evaluation Reforms and the Distribution of Teacher Effectiveness

In 2009, the New Teacher Project’s The Widget Effect documented the failure of U.S. public school districts to recognize and act on differences in teacher effectiveness. We revisit these findings by compiling teacher performance ratings across 24 states that adopted major reforms to their teacher evaluation systems. In the vast majority of these states, the percentage of teachers rated unsatisfactory remains less than 1%. However, the full distributions of ratings vary widely across states, with 0.7% to 28.7% rated below proficient and 6% to 62% rated above proficient. We present original survey data from an urban district illustrating that evaluators perceive more than 3 times as many teachers in their schools to be below proficient than they rate as such. Interviews with principals reveal several potential explanations for these patterns.

[1]  M. Lipsky,et al.  Street-Level Bureaucracy: The Dilemmas of the Individual in Public Service , 1983 .

[2]  Mark C. Glander.  Selected Statistics from the Public Elementary and Secondary Education Universe: School Year 2015-16. First Look. NCES 2018-052. , 2015 .

[3]  Pam Grossman,et al.  Measure for Measure: The Relationship between Measures of Instructional Practice in Middle School English Language Arts and Teachers' Value-Added Scores. NBER Working Paper No. 16015. , 2010 .

[4]  M. Kraft,et al.  The Sensitivity of Teacher Performance Ratings to the Design of Teacher Evaluation Systems , 2017 .

[5]  Michael Fullan,et al.  Choosing the wrong drivers for whole system reform , 2011 .

[6]  David Keeling,et al.  The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness. Second Edition. , 2009 .

[7]  M. Kraft,et al.  Can Principals Promote Teacher Development as Evaluators? A Case Study of Principals’ Views and Experiences , 2016, Educational administration quarterly : EAQ.

[8]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[9]  Tim R. Sass,et al.  Skills, Productivity and the Evaluation of Teacher Performance , 2012 .

[10]  S. Fine,et al.  Bringing values back in: How purposes shape practices in coherent school designs , 2015, Journal of Educational Change.

[11]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[12]  Matthew B. Miles,et al.  Qualitative Data Analysis: An Expanded Sourcebook , 1994 .

[13]  James P. Spillane,et al.  Journal of Educational Administration Emerald Article: School administration in a changing education sector: the US experience , 2012 .

[14]  E. Hanushek,et al.  Teachers, Schools, and Academic Achievement , 1998 .

[15]  Anthony S. Bryk,et al.  Finding a Measure of Trust@@@Trust in Schools: A Core Resource for Improvement , 2004 .

[16]  Jonah E. Rockoff,et al.  Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools , 2010 .

[17]  Jason A. Grissom,et al.  Make Room Value Added , 2015 .

[18]  I.,et al.  Street-Level Bureaucracy Revisited : Frontline District Central-Office Administrators as Boundary Spanners in Education Policy Implementation , 2022 .

[19]  R. Ingersoll,et al.  The Status of Teaching as a Profession , 2011 .

[20]  Jonah E. Rockoff.  The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data , 2004 .

[21]  Jason A. Grissom,et al.  Assessing Principals’ Assessments: Subjective Evaluations of Teacher Effectiveness in Low- and High-Stakes Environments , 2017, Education Finance and Policy.

[22]  Morgaen L. Donaldson,et al.  The New Educational Accountability: Understanding the Landscape of Teacher Evaluation in the Post-NCLB Era , 2016, Education Finance and Policy.

[23]  M. Engle.  Book Review: Qualitative Data Analysis: An Expanded Sourcebook (2nd Ed.) , 1999 .

[24]  Daniel F. McCaffrey,et al.  Have We Identified Effective Teachers? Validating Measures of Effective Teaching Using Random Assignment. Research Paper. MET Project. , 2013 .

[25]  R. Rothman,et al.  Rush to Judgment: Teacher Evaluation in Public Education. Education Sector Reports. , 2008 .

[26]  Morgaen L. Donaldson,et al.  Teacher Evaluation for Accountability and Development , 2014 .

[27]  Matthew G. Springer,et al.  The Impact of Performance Ratings on Job Satisfaction for Public School Teachers , 2017 .

[28]  Ronald H. Heck,et al.  Teacher evaluation and school improvement: An analysis of the evidence , 2014, Educational Assessment, Evaluation and Accountability.

[29]  Morgaen L. Donaldson.  Using Teacher Evaluation to Raise Teacher Quality , 2009 .

[30]  ohn,et al.  The Effect of Evaluation on Teacher Performance , 2011 .

[31]  Pamela D. Tucker.  Lake Wobegon: Where All Teachers Are Competent (Or, Have We Come to Terms with the Problem of Incompetent Teachers?) , 1997 .

[32]  Lauren Sartain,et al.  Does Teacher Evaluation Improve School Performance? Experimental Evidence from Chicago's Excellence in Teaching Project , 2015, Education Finance and Policy.

[33]  J. Papay.  Refocusing the Debate: Assessing the Purposes and Tools of Teacher Evaluation , 2012 .

[34]  Brian A. Jacob,et al.  Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education , 2008, Journal of Labor Economics.