Validation of a National Teacher Assessment and Improvement System

The task of validating a teacher assessment and improvement system is similar whether the system operates in the United States or in another country. Chile has a national teacher evaluation system (NTES) that is standards based, uses multiple instruments, and is intended to serve both formative and summative purposes. For the past 6 years the authors have performed validation research on NTES using a variety of methods and data sources. This article describes our validation research agenda, the results of major validation studies, and an integration of the existing evidence, and it offers the authors' preliminary judgment about NTES's validity. The article also offers a critical reflection regarding the decisions taken while driving the long and winding validation road, and the lessons we learned during this politically and methodologically complex journey.

[1]  Robert C. Pianta,et al.  An Argument Approach to Observation Protocol Validity , 2012 .

[2]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[3]  Hans Anand Pant,et al.  How Valid Are School Inspections? Problems and Strategies for Validating Processes and Results. , 2011 .

[4]  K. Rowe,et al.  Conceptualising and Evaluating Teacher Quality: Substantive and Methodological Issues , 2008 .

[5]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[6]  Allan S. Cohen,et al.  Validating Measures of Performance , 2005 .

[7]  J. Hattie,et al.  Assessing teachers for professional certification : the first decade of the National Board for Professional Teaching Standards , 2008 .

[8]  Kristopher J Preacher,et al.  Repairing Tom Swift's Electric Factor Analysis Machine , 2003 .

[9]  M. H. Elliott.  Assessing Accomplished Teaching AdvancedLevel Certification Programs , 2008 .

[10]  Heather C. Hill,et al.  A Validity Argument Approach to Evaluating Teacher Value-Added Scores , 2011 .

[11]  Nedjeljko Frančula.  The National Academies Press , 2013 .

[12]  I. Jolliffe.  Principal Component Analysis and Factor Analysis , 1986 .

[13]  S. Taut,et al.  Convergent validity evidence regarding the validity of the Chilean standards‐based teacher evaluation system , 2011 .

[14]  Lorrie A. Shepard,et al.  The Centrality of Test Use and Consequences for Test Validity. , 2005 .

[15]  E. Mooi,et al.  Principal Component and Factor Analysis , 2018, Springer Texts in Business and Economics.

[16]  Charlotte F. Danielson.  Enhancing Professional Practice: A Framework for Teaching , 1996 .

[17]  Brian M. Stecher,et al.  Perceived Effects of the Kentucky Instructional Results Information System (KIRIS). , 1996 .

[18]  Howard Eisner,et al.  Information Age , 2008, Wiley Encyclopedia of Computer Science and Engineering.

[19]  R. Linn.  Educational Accountability Systems. CSE Technical Report 687. , 2006 .

[20]  Brian M. Stecher,et al.  The Effects of the Washington State Education Reform on Schools and Classrooms. CSE Technical Report. , 2000 .

[21]  T. Brown,et al.  Confirmatory Factor Analysis for Applied Research , 2006 .

[22]  G. Engelhard,et al.  Examining the Psychometric Quality of the National Board for Professional Teaching Standards Early Childhood/Generalist Assessment System* , 2001 .

[23]  R. Correnti,et al.  Conceptual, Methodological, and Policy Issues in the Study of Teaching: Implications for Improving Instructional Practice at Scale , 2012 .

[24]  A. Strauss,et al.  Basics of Qualitative Research , 1992 .

[25]  John W. Creswell,et al.  Designing and Conducting Mixed Methods Research , 2006 .

[26]  A. Odden.  Lessons Learned About Standards-Based Teacher Evaluation Systems , 2004 .

[27]  Teaching Teachers: Professional Development To Improve Student Achievement , 2014 .

[28]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[29]  William S. Peters,et al.  Principal Components and Factor Analysis , 1987 .

[30]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[31]  Tina Trujillo,et al.  Corrective action in low performing schools: Lessons for NCLB implementation from first-generation accountability systems. , 2005 .

[32]  A. Anastasi.  Evolving Concepts of Test Validation , 1986 .

[33]  Christina Wikström,et al.  The concept of validity in theory and practice , 2010 .

[34]  Beatrice Ávalos,et al.  Moving from Resistance to Agreement: The Case of the Chilean Teacher Performance Evaluation. , 2006 .

[35]  山下 顕史,et al.  National board for professional teaching standardsの設立と現状に関する考察 , 2008 .

[36]  Martha Koch,et al.  Rethinking validation in complex high-stakes assessment contexts , 2012 .

[37]  E. Baker,et al.  Assessment Policy: Making Sense of the Babel , 2012 .

[38]  P. Moss.  Chapter 10 A critical review of the validity research agenda of the National Board for Professional Teaching Standards at the end of its first decade , 2004 .

[39]  Perceived effects and uses of the national teacher evaluation system in Chilean elementary schools , 2011 .

[40]  Samuel Messick,et al.  STANDARDS OF VALIDITY AND THE VALIDITY OF STANDARDS IN PERFORMANCE ASSESSMENT , 2005 .

[41]  John Hattie,et al.  Chapter 12 A validity study of the certification system of the National Board for Professional Teaching Standards , 2004 .

[42]  Joseph J. Pedulla.  National Council on Measurement in Education , 1984 .

[43]  Sandy Taut,et al.  Theory underlying a national teacher evaluation program. , 2010, Evaluation and program planning.

[44]  H. Heneman,et al.  Assessment of Teacher Reactions to a Standards-Based Teacher Evaluation System: A Pilot Study* , 2001 .

[45]  V. Santelices,et al.  La evaluación docente en Chile , 2011 .

[46]  D. Berliner.  The Near Impossibility of Testing for Teacher Quality , 2005 .

[47]  Sandy Taut,et al.  A mandatory, high-stakes National Teacher Evaluation System: Perceptions and attributions of teachers who actively refuse to participate , 2010 .

[48]  D. Koretz,et al.  Measuring up : what educational testing really tells us , 2008 .