Measuring Teaching Practices at Scale: A Novel Application of Text-as-Data Methods

Valid and reliable measurements of teaching quality facilitate school-level decision-making and policies pertaining to teachers, but conventional classroom observations are costly, prone to rater bias, and hard to implement at scale. Using nearly 1,000 word-to-word transcriptions of 4thand 5th-grade English language arts classes, we apply novel text-as-data methods to develop automated, objective measures of teaching to complement classroom observations. This approach is free of rater bias and enables the detection of three instructional factors that are well aligned with commonly used observation protocols: classroom management, interactive instruction, and teacher-centered instruction. The teacher-centered instruction factor is a consistent negative predictor of value-added scores, even after controlling for teachers’ average classroom observation scores. The interactive instruction factor predicts positive value-added scores.

[1]  Wolfgang Lörscher,et al.  Classroom discourse. The language of teaching and learning , 1990 .

[2]  S. Heath.  Ways with Words: Language, Life and Work in Communities and Classrooms , 1983 .

[3]  Neil Mercer,et al.  Teacher–Student Dialogue During Classroom Teaching: Does It Really Impact on Student Outcomes? , 2019, The Journal of the Learning Sciences.

[4]  Nathan D. Jones,et al.  Strategies for Assessing Classroom Teaching: Examining Administrator Thinking as Validity Evidence , 2018, Educational Assessment.

[5]  P. Carneiro,et al.  Teacher Quality and Learning Outcomes in Kindergarten , 2016, SSRN Electronic Journal.

[6]  Justin Grimmer,et al.  We Are All Social Scientists Now: How Big Data, Machine Learning, and Causal Inference Work Together , 2014, PS: Political Science & Politics.

[7]  Mary Budd Rowe,et al.  Wait Time: Slowing Down May Be A Way of Speeding Up! , 1986 .

[8]  Cindy K. Chung,et al.  When Small Words Foretell Academic Success: The Case of College Admissions Essays , 2014, PloS one.

[9]  Susan Goldin-Meadow,et al.  New Evidence about Language and Cognitive Development Based on a Longitudinal Study Hypotheses for Intervention Istrative and Technical Support. We Also Thank Varying the Language Learning Environment Susan Goldin- Meadow Varying the Language Learner Gesture: Another Perspective on Communicative Abi , 2022 .

[10]  Robert C. Pianta,et al.  Designing Teacher Evaluation Systems: New Guidance from the Measures of Effective Teaching Project , 2014 .

[11]  Jing Chen,et al.  Evaluating Efforts to Minimize Rater Bias in Scoring Classroom Observations , 2015 .

[12]  Helen F. Ladd,et al.  Teacher Credentials and Student Achievement: Longitudinal Analysis with Student Fixed Effects. , 2007 .

[13]  Margaret G. McKeown,et al.  Text Talk: Capturing the Benefits of Read-Aloud Experiences for Young Children. , 2001 .

[14]  J. Brophy.  Teacher influences on student achievement. , 1986 .

[15]  M. Nystrand.  Research on the Role of Classroom Discourse As It Affects Reading Comprehension , 2006 .

[16]  J. Allen,et al.  An Interaction-Based Approach to Enhancing Secondary School Instruction and Student Achievement , 2011, Science.

[17]  Patrick Donnelly,et al.  Automatically Measuring Question Authenticity in Real-World Classrooms , 2018 .

[18]  Richard C. Anderson,et al.  Patterns of discourse in two kinds of literature discussion , 2001 .

[19]  Robert C. Pianta,et al.  The Relation of Kindergarten Classroom Environment to Teacher, Family, and School Characteristics and Child Outcomes , 2002, The Elementary School Journal.

[20]  Jason A. Grissom,et al.  Assessing Principals’ Assessments: Subjective Evaluations of Teacher Effectiveness in Low- and High-Stakes Environments , 2017, Education Finance and Policy.

[21]  Daniel A. McFarland,et al.  Making the Connection: Social Bonding in Courtship Situations1 , 2013, American Journal of Sociology.

[22]  David M. Blei,et al.  Probabilistic topic models , 2012, Commun. ACM.

[23]  Jonah E. Rockoff,et al.  Teacher Applicant Hiring and Teacher Performance: Evidence from Dc Public Schools , 2016, Journal of Public Economics.

[24]  J. Pennebaker,et al.  Linguistic Style Matching in Social Interaction , 2002 .

[25]  Jing Liu,et al.  Differing Views of Equity: How Prospective Educators Perceive Their Role in Closing Achievement Gaps , 2019, RSF.

[26]  Charlotte Danielson,et al.  Evaluations that Help Teachers Learn. , 2011 .

[27]  Laura M. Justice,et al.  Teacher-child conversations in preschool classrooms: Contributions to children's vocabulary development , 2015 .

[28]  D. Ball,et al.  Unpacking Pedagogical Content Knowledge: Conceptualizing and Measuring Teachers' Topic-Specific Knowledge of Students , 2008, Journal for Research in Mathematics Education.

[29]  Brian Gill,et al.  The Content, Predictive Power, and Potential Bias in Five Widely Used Teacher Observation Instruments. REL 2017-191. , 2016 .

[30]  R. Gallimore,et al.  Rousing Minds to Life: Teaching, Learning, and Schooling in Social Context , 1988 .

[31]  T. Seidel,et al.  Teaching Effectiveness Research in the Past Decade: The Role of Theory and Research Design in Disentangling Meta-Analysis Results , 2007 .

[32]  Andrew J. Mashburn,et al.  Measures of classroom quality in prekindergarten and children's development of academic, language, and social skills. , 2008, Child development.

[33]  Kathy Hirsh-Pasek,et al.  A matter of principle: Applying language science to the classroom and beyond. , 2017 .

[34]  Jesse Rothstein,et al.  Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement , 2008 .

[35]  Tricia A. Zucker,et al.  Shared-reading dynamics: mothers' question use and the verbal participation of children with specific language impairment. , 2012, Journal of speech, language, and hearing research : JSLHR.

[36]  Martin Nystrand,et al.  Discussion-Based Approaches to Developing Understanding: Classroom Instruction and Student Performance in Middle and High School English , 2003 .

[37]  Pam Grossman,et al.  Measure for Measure: The Relationship between Measures of Instructional Practice in Middle School English Language Arts and Teachers' Value-Added Scores. NBER Working Paper No. 16015. , 2010 .

[38]  Jure Leskovec,et al.  Large-scale Analysis of Counseling Conversations: An Application of Natural Language Processing to Mental Health , 2016, TACL.

[39]  John Nerbonne,et al.  The Secret Life of Pronouns. What Our Words Say About Us , 2014, Lit. Linguistic Comput..

[40]  Jing Liu,et al.  Connections Matter: How Interactive Peers Affect Students in Online College Courses , 2016 .

[41]  L. S. Vygotskiĭ,et al.  Mind in society : the development of higher psychological processes , 1978 .

[42]  Dan Goldhaber,et al.  Building a More Complete Understanding of Teacher Evaluation Using Classroom Observations , 2016 .

[43]  E. Hanushek,et al.  Teachers, Schools, and Academic Achievement , 1998 .

[44]  Cristian Danescu-Niculescu-Mizil,et al.  Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions , 2016, WWW.

[45]  J. Huttenlocher,et al.  Language input and language growth. , 1998, Preventive medicine.

[46]  Virginia Richardson,et al.  On Making Determinations of Quality in Teaching. , 2005 .

[47]  Cory Koedel,et al.  Value-added modeling: A review , 2015 .

[48]  J. Allen,et al.  Observations of Effective Teacher–Student Interactions in Secondary School Classrooms: Predicting Student Achievement With the Classroom Assessment Scoring System—Secondary , 2013, School psychology review.

[49]  ohn,et al.  The Effect of Evaluation on Teacher Performance , 2011 .

[50]  C. Tamis-LeMonda,et al.  Reciprocal influences between maternal language and children's language and cognitive development in low-income families* , 2013, Journal of Child Language.

[51]  Charalambos Y. Charalambous,et al.  When Rater Reliability Is Not Enough , 2012 .

[52]  David Blazar,et al.  Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement , 2015 .

[53]  Kevin F. Miller,et al.  Using the LENA in teacher training: Promoting student involvement through automated feedback , 2013 .

[54]  Annemarie S. Palincsar,et al.  Enhancing Instructional Time Through Attention to Metacognition , 1987, Journal of learning disabilities.

[55]  S. Graham,et al.  Self-Regulated Strategy Development: Helping Students with Learning Problems Develop as Writers , 1993, The Elementary School Journal.

[56]  John H. Tyler,et al.  Identifying Effective Classroom Practices Using Student Achievement Data , 2010 .

[57]  Matthew A. Kraft,et al.  Revisiting The Widget Effect: Teacher Evaluation Reforms and the Distribution of Teacher Effectiveness , 2017 .

[58]  Margaret H. Szymanski,et al.  LEARNING IN DOING: SOCIAL, COGNITIVE AND COMPUTATIONAL PERSPECTIVES , 2011 .

[59]  Cristian Danescu-Niculescu-Mizil,et al.  Conversational Markers of Constructive Discussions , 2016, NAACL.

[60]  B. Hamre,et al.  Rater calibration when observational assessment occurs at large scale: Degree of calibration and characteristics of raters associated with calibration , 2012 .

[61]  Min Sun,et al.  Using a Text-as-Data Approach to Understand Reform Processes: A Deep Exploration of School Improvement Strategies , 2019, Educational Evaluation and Policy Analysis.

[62]  S. Michaels,et al.  Deliberative Discourse Idealized and Realized: Accountable Talk in the Classroom and in Civic Life , 2008 .

[63]  P. David Pearson,et al.  Looking inside classrooms: Reflecting on the “how” as well as the “what” in effective reading instruction. , 2002 .

[64]  N. Gage,et al.  Process-Product Research on Teaching: A Review of Criticisms , 1989, The Elementary School Journal.

[65]  M. Nystrand,et al.  Instructional Discourse, Student Engagement, and Literature Achievement , 1991, Research in the Teaching of English.

[66]  David Keeling,et al.  The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness. Second Edition. , 2009 .

[67]  Andrew D. Ho,et al.  The Reliability of Classroom Observations by School Personnel. Research Paper. MET Project. , 2013 .

[68]  B. Hamre,et al.  Early teacher-child relationships and the trajectory of children's school outcomes through eighth grade. , 2001, Child development.

[69]  Raymond L. Pecheone,et al.  Gathering Feedback for Teaching Combining High-Quality Observations with Student Surveys and Achievement Gains , 2012 .

[70]  Robert C. Pianta,et al.  An Argument Approach to Observation Protocol Validity , 2012 .

[71]  H. Mehan.  ‘What time is it, Denise?”: Asking known information questions in classroom discourse , 1979 .

[72]  Faye L. Mueller,et al.  Apprenticing Adolescent Readers to Academic Literacy , 2001 .