Evaluating the quality of teaching: can there be valid differentiation in the middle of the performance distribution?

ABSTRACT An important challenge in evaluating the quality of teaching is to arrive at valid distinctions between teachers, particularly in the middle of the performance distribution. This study addressed this issue for the Chilean national teacher evaluation system. Mathematics teachers (N = 51) from the 2 middle adjacent performance categories, and their students, were examined by videotaping classroom practice and coding the videos with the Classroom Assessment Scoring System for Secondary School, measuring students’ perceptions of their teachers’ practices and assessing student learning on standardized grade-level and unit-specific tests. Findings show that on almost all criterion variables, competent teachers outperformed basic teachers, but differences were usually small. Teachers evaluated as competent obtained consistently better results regarding their teaching practices, as assessed by trained observers, as well as based on reports by their own students. However, there was only weak indication of differences between the 2 groups in terms of student learning.

[1]  W. Hays.  Using Multivariate Statistics , 1983 .

[2]  Convergent validity evidence regarding the validity of the Chilean standards‐based teacher evaluation system , 2011 .

[3]  L. Goe,et al.  Teacher Evaluation in Chile 2013 , 2013 .

[4]  J. Hattie,et al.  Assessing teachers for professional certification : the first decade of the National Board for Professional Teaching Standards , 2008 .

[5]  John Hattie,et al.  Identifying accomplished teachers: a validation study. , 2008 .

[6]  J. Scheerens.  Theories on educational effectiveness and ineffectiveness , 2015, Educational Effectiveness Theory.

[7]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[8]  Courtney A. Bell,et al.  The Instructional Challenge in Improving Teaching Quality: Lessons from a Classroom Observation Protocol , 2014, Teachers College Record: The Voice of Scholarship in Education.

[9]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[10]  M. Isoré,et al.  Teacher Evaluation: Current Practices in OECD Countries and a Literature Review , 2009 .

[11]  Charlotte F. Danielson.  Enhancing Professional Practice: A Framework for Teaching , 1996 .

[12]  Raymond L. Pecheone,et al.  Learning about Teaching Initial Findings from the Measures of Effective Teaching Project , 2010 .

[13]  J. Allen,et al.  An Interaction-Based Approach to Enhancing Secondary School Instruction and Student Achievement , 2011, Science.

[14]  Beatrice Ávalos,et al.  Moving from Resistance to Agreement: The Case of the Chilean Teacher Performance Evaluation. , 2006 .

[15]  B. Lazerick.  Third international mathematics and science study , 1997 .

[16]  Brian M. Stecher,et al.  Validation of a National Teacher Assessment and Improvement System , 2012 .

[17]  Thomas R. Boucher,et al.  Test Equating, Scaling, and Linking: Methods and Practices , 2007 .

[18]  Anthony S. Bryk,et al.  Hierarchical Linear Models: Applications and Data Analysis Methods , 1992 .

[19]  T. Seidel,et al.  Teaching Effectiveness Research in the Past Decade: The Role of Theory and Research Design in Disentangling Meta-Analysis Results , 2007 .

[20]  Anthony Milanowski.  The Relationship Between Teacher Performance Evaluation Scores and Student Achievement: Evidence From Cincinnati , 2004 .

[21]  J. Durlak.  How to select, calculate, and interpret effect sizes. , 2009, Journal of pediatric psychology.

[22]  S. Taut,et al.  Teacher performance and student learning: linking evidence from two national assessment programmes , 2016, Educational Assessment in Latin America.

[23]  Paulo Santiago,et al.  Teachers matter : Attracting, Developing and Retaining Effective Teachers , 2005 .

[24]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[25]  John Hattie,et al.  Chapter 11 Identifying accomplished teachers: A validation study , 2004 .

[26]  Robert C. Pianta,et al.  Teaching through Interactions , 2013, The Elementary School Journal.

[27]  Pam Sammons,et al.  The dynamics of educational effectiveness: a contribution to policy, practice and theory in contemporary schools , 2009 .

[28]  L. Kyriakides,et al.  Critical analysis of the current approaches to modelling educational effectiveness: The importance of establishing a dynamic model , 2006 .

[29]  Clifton Bob Clark,et al.  The University of North Carolina at Greensboro , 1980 .

[30]  Lee J. Cronbach,et al.  Construct validation after thirty years. , 1989 .

[31]  J. Allen,et al.  Observations of Effective Teacher–Student Interactions in Secondary School Classrooms: Predicting Student Achievement With the Classroom Assessment Scoring System—Secondary , 2013, School psychology review.

[32]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[33]  Harvey Goldstein,et al.  Multilevel Statistical Models: Goldstein/Multilevel Statistical Models , 2010 .

[34]  B. Hamre,et al.  Learning opportunities in preschool and early elementary classrooms. , 2007 .

[35]  Karen B. Givvin,et al.  Third International Mathematics and Science Study 1999 Video Study Technical Report , 2003 .

[36]  Sandy Taut,et al.  The Development and Implementation of a National, Standards-based, Multi-method Teacher Performance Assessment System in Chile , 2014 .

[37]  B. Schmitz,et al.  Student and teacher ratings of instructional quality: Consistency of ratings over time, agreement, and predictive power. , 2016 .

[38]  L. Hedges,et al.  How Large Are Teacher Effects? , 2004 .

[39]  Robert C. Pianta,et al.  An Argument Approach to Observation Protocol Validity , 2012 .

[40]  Ministerio de Educación,et al.  Marco para la buena enseñanza , 2018 .

[41]  Courtney A. Bell,et al.  Evaluating teaching and teachers. , 2013 .

[42]  M. Kennedy.  Teacher assessment and the quest for teacher quality : a handbook , 2014 .

[43]  D. Eignor.  The standards for educational and psychological testing. , 2013 .

[44]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[45]  Daniel D. Liou,et al.  Asian American Education and Income Attainment in the Era of Post-Racial America , 2014, Teachers College Record: The Voice of Scholarship in Education.

[46]  Ministerio de Educación..  Orientaciones para la apropiación de las Bases Curriculares 7° Básico a 2° Medio. , 2017 .

[47]  Robert C. Pianta,et al.  Teaching Through Interactions in Secondary School Classrooms , 2014, The Journal of early adolescence.

[48]  Eckhard Klieme,et al.  Student ratings of teaching quality in primary school: Dimensions and prediction of student outcomes , 2014 .