Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes

Mathematics professional development is widely offered, typically with the goal of improving teachers’ content knowledge, the quality of teaching, and ultimately students’ achievement. Recently, new assessments focused on mathematical knowledge for teaching (MKT) have been developed to assist in the evaluation and improvement of mathematics professional development. This study presents empirical estimates of average program change in MKT and its variation with the goal of supporting the design of experimental trials that are adequately powered to detect a specified program effect. The study drew on a large database representing five different assessments of MKT and collectively 326 professional development programs and 9,365 teachers. Results from cross-classified hierarchical growth models found that standardized average change estimates across the five assessments ranged from a low of 0.16 standard deviations (SDs) to a high of 0.26 SDs. Power analyses using the estimated pre- and posttest change estimates indicated that hundreds of teachers are needed to detect changes in knowledge at the lower end of the distribution. Even studies powered to detect effects at the higher end of the distribution will require substantial resources to conduct rigorous experimental trials. Empirical benchmarks that describe average program change and its variation provide a useful preliminary resource for interpreting the relative magnitude of effect sizes associated with professional development programs and for designing adequately powered trials.

[1]  L. Hedges,et al.  How Large Are Teacher Effects? , 2004 .

[2]  Deborah Loewenberg Ball,et al.  Content Knowledge for Teaching , 2008 .

[3]  Laura M. Desimone,et al.  Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study , 2002 .

[4]  Ben Kelcey,et al.  Considerations for Designing Group Randomized Trials of Professional Development With Teacher Knowledge Outcomes , 2013 .

[5]  Jonah E. Rockoff,et al.  provided that full credit, including © notice, is given to the source. Can You Recognize an Effective Teacher When You Recruit One? , 2008 .

[6]  H. Bloom,et al.  New Empirical Evidence for the Design of Group Randomized Trials in Education , 2010 .

[7]  H. Borko.  Professional Development and Teacher Learning: Mapping the Terrain , 2004 .

[8]  Nicole B. Kersting,et al.  Using Video Clips of Mathematics Classroom Instruction as Item Prompts to Measure Teachers' Knowledge of Teaching Mathematics , 2008 .

[9]  James W. Stigler,et al.  Teachers’ Analyses of Classroom Video Predict Student Learning of Mathematics: Further Explorations of a Novel Measure of Teacher Knowledge , 2010 .

[10]  R. Brennan,et al.  Test Equating, Scaling, and Linking , 2004 .

[11]  D. Gitomer,et al.  Developing Content Knowledge for Teaching Assessments for the Measures of Effective Teaching Study. Research Report. ETS RR-14-33. , 2014 .

[12]  S. Krauss,et al.  Secondary mathematics teachers’ pedagogical content knowledge and content knowledge: validation of the COACTIV constructs , 2008 .

[13]  S. Krauss,et al.  Teachers’ Mathematical Knowledge, Cognitive Activation in the Classroom, and Student Progress , 2010 .

[14]  Jennifer I. Berne,et al.  Chapter 6 : Teacher Learning and the Acquisition of Professional Knowledge: An Examination of Research on Contemporary Professlonal Development , 1999 .

[15]  Stephanie Cronen,et al.  Experimenting With Teacher Professional Development: Motives and Methods , 2008 .

[16]  Laura M. Desimone,et al.  What Makes Professional Development Effective? Results From a National Sample of Teachers , 2001 .

[17]  Ben Kelcey,et al.  Strategies for Improving Power in School-Randomized Studies of Professional Development , 2013, Evaluation review.

[18]  Mark W. Lipsey,et al.  Empirical Benchmarks for Interpreting Effect Sizes in Research , 2008 .

[19]  Heather C. Hill,et al.  Effects of Teachers’ Mathematical Knowledge for Teaching on Student Achievement , 2005 .

[20]  James W. Stigler,et al.  Measuring Usable Knowledge , 2012 .

[21]  H. Hill,et al.  Professional Development Research , 2013 .

[22]  Peter Z. Schochet.  Statistical Power for Random Assignment Evaluations of Education Programs , 2005 .

[23]  Charalambos Y. Charalambous,et al.  Mathematical Knowledge for Teaching and the Mathematical Quality of Instruction: An Exploratory Study , 2008 .

[24]  D. Ball,et al.  Developing Measures of Teachers’ Mathematics Knowledge for Teaching , 2004, The Elementary School Journal.

[25]  Laura M. Desimone,et al.  Improving Impact Studies of Teachers’ Professional Development: Toward Better Conceptualizations and Measures , 2009 .

[26]  Ralph T. Putnam,et al.  Teacher Learning: Implications of New Views of Cognition , 1997 .