An Argument Approach to Observation Protocol Validity

This article develops a validity argument approach for use on observation protocols currently used to assess teacher quality for high-stakes personnel and professional development decisions. After defining the teaching quality domain, we articulate an interpretive argument for observation protocols. To illustrate the types of evidence that might compose a validity argument, we draw on data from a validity study of the Classroom Assessment Scoring System for secondary classrooms. Based on data from 82 Algebra classrooms, we illustrate how data from observation scores, value-added models, generalizability studies, and measures of teacher knowledge, student achievement, and teacher and student beliefs could be used to build a validity argument for observation protocols. Strengths and limitations of the validity argument approach as well as the issues the approach raises for observation protocol validity research are considered.

[1]  Robert C. Pianta,et al.  Teaching through Interactions , 2013, The Elementary School Journal.

[2]  Courtney A. Bell,et al.  Effect of Observation Mode on Measures of Secondary Mathematics Teaching , 2013 .

[3]  Raymond L. Pecheone,et al.  Gathering Feedback for Teaching Combining High-Quality Observations with Student Surveys and Achievement Gains , 2012 .

[4]  J. Allen,et al.  An Interaction-Based Approach to Enhancing Secondary School Instruction and Student Achievement , 2011, Science.

[5]  Heather C. Hill,et al.  A Validity Argument Approach to Evaluating Teacher Value-Added Scores , 2011 .

[6]  David Keeling,et al.  The Widget Effect: Our National Failure to Acknowledge and Act on Differences in Teacher Effectiveness. Second Edition. , 2009 .

[7]  Kevin J Grimm,et al.  The contribution of children's self-regulation and classroom quality to children's adaptive behaviors in the kindergarten classroom. , 2009, Developmental psychology.

[8]  Deborah Loewenberg Ball,et al.  Content Knowledge for Teaching , 2008 .

[9]  Charalambos Y. Charalambous,et al.  Mathematical Knowledge for Teaching and the Mathematical Quality of Instruction: An Exploratory Study , 2008 .

[10]  Courtney A. Bell,et al.  Approaches to Evaluating Teacher Effectiveness: A Research Synthesis. , 2008 .

[11]  Robert C. Pianta,et al.  Ready to learn? Children's pre-academic achievement in pre-Kindergarten programs , 2008 .

[12]  L. Goe.  The Link between Teacher Quality and Student Outcomes: A Research Synthesis. , 2007 .

[13]  Heather C. Hill,et al.  Validating the Ecological Assumption: The Relationship of Measure Scores to Classroom Teaching and Student Learning , 2007 .

[14]  Daniel F. McCaffrey,et al.  The Sensitivity of Value‐Added Teacher Effect Estimates to Different Mathematics Achievement Measures , 2007 .

[15]  Lisa S. Blackwell,et al.  Implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and an intervention. , 2007, Child development.

[16]  J. R. Lockwood,et al.  Using Structured Classroom Vignettes to Measure Instructional Practices in Mathematics , 2006 .

[17]  B. Hamre,et al.  Can instructional and emotional support in the first-grade classroom make a difference for children at risk of school failure? , 2005, Child development.

[18]  R. Pianta,et al.  Features of Pre-Kindergarten Programs, Classrooms, and Teachers: Do They Predict Observed Classroom Quality and Child-Teacher Interactions? , 2005 .

[19]  S. Rimm-Kaufman,et al.  The Contribution of Classroom Setting and Quality of Instruction to Children’s Behavior in Kindergarten Classrooms , 2005, The Elementary School Journal.

[20]  H. Cooper,et al.  The Relation Between Self-Beliefs and Academic Achievement: A Meta-Analytic Review , 2004 .

[21]  Robert C. Pianta,et al.  The Classroom Assessment Scoring System: Findings from the Prekindergarten Year , 2004, The Elementary School Journal.

[22]  Nigel O'Brian,et al.  Generalizability Theory I , 2003 .

[23]  Peter A. Youngs,et al.  Teacher Characteristics and Student Achievement Gains: A Review , 2003 .

[24]  Kathleen Vail,et al.  Ready To Learn. , 2003 .

[25]  James P. Spillane,et al.  Reform and Teaching: Exploring Patterns of Practice in the Context of National and State Mathematics Reforms , 1999 .

[26]  R. Jaeger.  Live vs. Memorex: Psychometric and Practical Issues in the Collection of Data on Teachers' Performances in the Classroom. , 1993 .

[27]  M. Pajares,et al.  Teachers’ Beliefs and Educational Research: Cleaning Up a Messy Construct , 1992 .

[28]  Richard J. Shavelson,et al.  Generalizability Theory: A Primer , 1991 .

[29]  D. Cicchetti,et al.  Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. , 1981, American journal of mental deficiency.

[30]  Eva Nick,et al.  The dependability of behavioral measurements: theory of generalizability for scores and profiles , 1973 .

[31]  Donald B. Rubin,et al.  The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. , 1974 .

[32]  Henry Beaumont,et al.  The Measurement of Teaching , 1938 .