Inferring constructs of effective teaching from classroom observations: An application of Bayesian exploratory factor analysis without restrictions

Ratings of teachers' instructional practices using standardized classroom observation instruments are increasingly being used for both research and teacher accountability. There are multiple instruments in use, each attempting to evaluate many dimensions of teaching and classroom activities, and little is known about what underlying teaching quality attributes are being measured. We use data from multiple instruments collected from 458 middle school mathematics and English language arts teachers to inform research and practice on teacher performance measurement by modeling latent constructs of high-quality teaching. We make inferences about these constructs using a novel approach to Bayesian exploratory factor analysis (EFA) that, unlike commonly used approaches for identifying factor loadings in Bayesian EFA, is invariant to how the data dimensions are ordered. Applying this approach to ratings of lessons reveals two distinct teaching constructs in both mathematics and English language arts: (1) quality of instructional practices; and (2) quality of teacher management of classrooms. We demonstrate the relationships of these constructs to other indicators of teaching quality, including teacher content knowledge and student performance on standardized tests.

[1]  Charlotte F. Danielson.  Enhancing Professional Practice: A Framework for Teaching , 1996 .

[2]  David B Dunson,et al.  Default Prior Distributions and Efficient Posterior Computation in Bayesian Factor Analysis , 2009, Journal of computational and graphical statistics : a joint publication of American Statistical Association, Institute of Mathematical Statistics, Interface Foundation of North America.

[3]  H. Kaiser.  The Application of Electronic Computers to Factor Analysis , 1960 .

[4]  Daniel F McCaffrey,et al.  Bayesian Hierarchical Multivariate Formulation with Factor Analysis for Nested Ordinal Data , 2013, Psychometrika.

[5]  Hedibert Freitas Lopes,et al.  Parsimonious Bayesian Factor Analysis when the Number of Factors is Unknown , 2010 .

[6]  Peter Congdon,et al.  Bayesian Models for Categorical Data , 2005 .

[7]  Michael A. West,et al.  BAYESIAN MODEL ASSESSMENT IN FACTOR ANALYSIS , 2004 .

[8]  A. Gelman.  Prior distributions for variance parameters in hierarchical models (comment on article by Browne and Draper) , 2004 .

[9]  Valen E. Johnson,et al.  On Bayesian Analysis of Multirater Ordinal Data: An Application to Automated Essay Grading , 1996 .

[10]  Pam Grossman,et al.  Measure for Measure: The Relationship between Measures of Instructional Practice in Middle School English Language Arts and Teachers' Value-Added Scores. NBER Working Paper No. 16015. , 2010 .

[11]  J. R. Lockwood,et al.  Trends in Classroom Observation Scores , 2015, Educational and psychological measurement.

[12]  Robert C. Pianta,et al.  Teaching through Interactions , 2013, The Elementary School Journal.

[13]  R. Hambleton,et al.  Handbook of Modern Item Response Theory , 1997 .

[14]  C. Carvalho,et al.  A sparse factor analytic probit model for congressional voting patterns , 2012 .

[15]  Thomas J. Kane,et al.  Identifying Effective Teachers Using Performance on the Job. The Hamilton Project Policy Brief No. 2006-01. , 2006 .

[16]  H. Kaiser.  The varimax criterion for analytic rotation in factor analysis , 1958 .

[17]  Courtney A. Bell,et al.  The Instructional Challenge in Improving Teaching Quality: Lessons from a Classroom Observation Protocol , 2014, Teachers College Record: The Voice of Scholarship in Education.

[18]  Thomas J. Kane,et al.  Ensuring Fair and Reliable Measures of Effective Teaching Culminating Findings from the MET Project ’ s Three-Year Study , 2013 .

[19]  M. West,et al.  High-Dimensional Sparse Factor Modeling: Applications in Gene Expression Genomics , 2008, Journal of the American Statistical Association.

[20]  L. Shulman.  Knowledge and Teaching: Foundations of the New Reform , 1987 .

[21]  S. Geisser,et al.  A Predictive Approach to Model Selection , 1979 .

[22]  J. Horn.  A rationale and test for the number of factors in factor analysis , 1965, Psychometrika.

[23]  M. Browne.  An Overview of Analytic Rotation in Exploratory Factor Analysis , 2001 .

[24]  C. Morris.  Parametric Empirical Bayes Inference: Theory and Applications , 1983 .

[25]  Martyn Plummer,et al.  JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling , 2003 .

[26]  Gaetano Carmeci.  A Metropolis-Hastings algorithm for reduced rank covariance matrices with application to Bayesian factor models , 2008 .

[27]  Elena A. Erosheva,et al.  Dealing with rotational invariance in Bayesian conrmatory factor analysis , 2013 .

[28]  J. Hartigan,et al.  The Dip Test of Unimodality , 1985 .

[29]  Damien McParland,et al.  CLUSTERING SOUTH AFRICAN HOUSEHOLDS BASED ON THEIR ASSET STATUS USING LATENT VARIABLE MODELS. , 2014, The annals of applied statistics.

[30]  Robert C. Pianta,et al.  A Course on Effective Teacher-Child Interactions , 2012 .

[31]  Robert Gordon,et al.  Identifying Effective Teachers Using Performance on the Job. Discussion Paper 2006-01. , 2006 .

[32]  Peter D. Hoff,et al.  Latent Space Approaches to Social Network Analysis , 2002 .

[33]  M. Stephens.  Dealing with label switching in mixture models , 2000 .

[34]  J. Geweke,et al.  Measuring the pricing error of the arbitrage pricing theory , 1996 .

[35]  H. Ishwaran.  Univariate and multirater ordinal cumulative link regression with covariate specific cutpoints , 2000 .

[36]  Terrance D. Savitsky,et al.  Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors , 2015 .

[37]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[38]  D. Dunson,et al.  Sparse Bayesian infinite factor models. , 2011, Biometrika.

[39]  James G. Scott,et al.  The horseshoe estimator for sparse signals , 2010 .

[40]  Daniel F. McCaffrey,et al.  Correcting for Test Score Measurement Error in ANCOVA Models for Estimating Treatment Effects , 2014 .