Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Although scalable programs, such as online courses, have the potential to reach broad audiences, they may pose challenges to evaluating learners’ knowledge and skills. Automated scoring offers a possible solution. In the current paper, we describe the process of creating and testing an automated means of scoring a validated measure of teachers’ observational skills, known as the Video Assessment of Instructional Learning (VAIL). Findings show that automated VAIL scores were consistently correlated with scores assigned by the hand scoring system. In addition, the automated VAIL replicated intervention effects found in the hand scoring system. The automated scoring technique appears to offer an efficient and reliable assessment. This study may offer additional insight into how to utilize similar techniques in other large-scale programs and interventions.

[1]  S. Gest,et al.  Individual Factors Associated With Professional Development Training Outcomes of the Head Start REDI Program , 2009, Early education and development.

[2]  Les Perelman,et al.  When “the state of the art” is counting words , 2014 .

[3]  Chris Brew,et al.  Automated Short Answer Scoring: Principles and Prospects , 2013 .

[4]  C. Boston.  The Concept of Formative Assessment. , 2002 .

[5]  W. Foddy.  Constructing Questions for Interviews and Questionnaires: Theory and Practice in Social Research , 1993 .

[6]  William Condon,et al.  Large-Scale Assessment, Locally-Developed Measures, and Automated Scoring of Essays: Fishing for Red Herrings? , 2013 .

[7]  Andrew J. Mashburn,et al.  Effects of Web-Mediated Professional Development Resources on Teacher-Child Interactions in Pre-Kindergarten Classrooms. , 2008, Early childhood research quarterly.

[8]  Semire Dikli,et al.  An Overview of Automated Scoring of Essays. , 2006 .

[9]  M. Greenberg,et al.  Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program , 2008, Development and Psychopathology.

[10]  William T. Gormley,et al.  Investing in Our Future: The Evidence Base on Preschool Education , 2013 .

[11]  Robert C. Pianta,et al.  A Course on Effective Teacher-Child Interactions , 2012 .

[12]  Martin Chodorow,et al.  CriterionSM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays , 2003, IAAI.

[13]  Jill Burstein,et al.  Handbook of Automated Essay Evaluation Current Applications and New Directions , 2018 .

[14]  G. Boulton‐Lewis.  Teaching for quality learning at university , 2008 .

[15]  Allison Littlejohn,et al.  Massive Open Online Courses: a traditional or transformative approach to learning? , 2014 .

[16]  Rena M. Palloff,et al.  Assessing the Online Learner: Resources and Strategies for Faculty , 2008 .

[17]  R. Pianta,et al.  Predicting Child Outcomes at the End of Kindergarten from the Quality of Pre-Kindergarten Teacher–Child Interactions and Instruction , 2008 .

[18]  J. DeCoster,et al.  Using a Standardized Video-Based Assessment in a University Teacher Education Program to Examine Preservice Teachers Knowledge Related to Effective Teaching. , 2013 .

[19]  B. Hamre,et al.  Classroom assessment scoring system. , 2008 .

[20]  Randolph A. Philipp,et al.  SITUATION AWARENESS IN TEACHING: What Educators Can Learn From Video-Based Research in Other Fields , 2011 .

[21]  Wanda E. Gill,et al.  The Ready to Teach Program: A Federal Initiative in Support of Online Courses for Teachers. , 2011 .

[22]  Martin Chodorow,et al.  C-rater: Automated Scoring of Short-Answer Questions , 2003, Comput. Humanit..

[23]  Kyle M. Williams,et al.  Global Times Call for Global Measures: Investigating Automated Essay Scoring in Linguistically-Diverse MOOCs. , 2016 .

[24]  T. Landauer.  Automatic Essay Assessment , 2003 .

[25]  David M. Williamson,et al.  A Framework for Evaluation and Use of Automated Scoring , 2012 .

[26]  Amy C. Thomason,et al.  Measuring the Quality of Teacher–Child Interactions in Toddler Child Care , 2009 .

[27]  David M. Williamson,et al.  Automated essay scoring: Psychometric guidelines and practices , 2013 .

[28]  B. Hamre,et al.  Assessing Teachers’ Skills in Detecting and Identifying Effective Interactions in the Classroom , 2015, The Elementary School Journal.

[29]  A. Bandura.  Social Foundations of Thought and Action: A Social Cognitive Theory , 1985 .

[30]  Barbara Means,et al.  Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies , 2009 .

[31]  James W. Pennebaker,et al.  Linguistic Inquiry and Word Count (LIWC2007) , 2007 .