Formative Evaluation

Performance-based accountability along with budget tightening has increased pressure on publicly funded organizations to develop and deliver programs that produce meaningful social benefits. As a result, there is increasing need to undertake formative evaluations that estimate preliminary program outcomes and identify promising program components based on their effectiveness during implementation. By combining longitudinal administrative data, multiple comparison group designs, and a progressive series of analyses that test rival explanations, evaluators can strengthen causal arguments and provide actionable program information for key stakeholders to improve program outcomes. In this article, we illustrate the application of rigorous methods to estimate preliminary program effects and rule out alternative explanations for preliminary effects, including site selection bias, individual selection bias, and resentful demoralization through the evaluation of the Collaborative Project, a North Carolina educational improvement project that incorporated multiple components aimed at boosting student achievement.

[1]  Susanna Loeb,et al.  How Changes in Entry Requirements Alter the Teacher Workforce and Affect Student Achievement , 2005, Education Finance and Policy.

[2]  Robert F. Testa,et al.  Educational Research: Competencies for Analysis and Application , 1979 .

[3]  Gary T. Henry,et al.  Teacher Preparation Policies and Their Effects on Student Achievement , 2014, Education Finance and Policy.

[4]  Peter G. Polson,et al.  The Consequences of Consistent and Inconsistent User Intertaces , 2013 .

[5]  Kevin C. Bastian,et al.  The Effects of Experience and Attrition for Novice High-School Science and Mathematics Teachers , 2012, Science.

[6]  Martin Tessmer.  Formative Evaluation Alternatives. , 2008 .

[7]  Aliza Duby.  Early Formative Evaluation of Educational Television , 1988 .

[8]  Juan-José Díaz,et al.  An Assessment of Propensity Score Matching as a Nonexperimental Impact Estimator , 2005, The Journal of Human Resources.

[9]  H. Krumholz,et al.  Quality improvement studies: the need is there but so are the challenges. , 2000, The American journal of medicine.

[10]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[11]  N. Denzin,et al.  Strategies Of Qualitative Inquiry , 2012 .

[12]  D. Rubin,et al.  Principal Stratification in Causal Inference , 2002, Biometrics.

[13]  R. Bifulco,et al.  CAN NONEXPERIMENTAL ESTIMATES REPLICATE ESTIMATES BASED ON RANDOM ASSIGNMENT IN EVALUATIONS OF SCHOOL CHOICE? A WITHIN-STUDY COMPARISON , 2012 .

[14]  Steven Glazerman,et al.  Nonexperimental Versus Experimental Estimates of Earnings Impacts , 2003 .

[15]  G. Ohlin.  The Organization for Economic Cooperation and Development , 1968, International Organization.

[16]  A. R. Ilersic,et al.  Research methods in social relations , 1961 .

[17]  Joshua D. Angrist,et al.  Identification of Causal Effects Using Instrumental Variables , 1993 .

[18]  D. Caulley.  Qualitative research for education: An introduction to theories and methods , 2007 .

[19]  Richard A. Berk,et al.  Police Responses to Family Violence Incidents: An Analysis of an Experimental Design with Incomplete Randomization , 1988 .

[20]  Tom Wujec,et al.  Multimedia Interface Design , 1993, ICHIM.

[21]  I. Seidman.  Interviewing as qualitative research : a guide for researchersin education and the social sciences , 1991 .

[22]  D. Brewer,et al.  Does Teacher Certification Matter? High School Teacher Certification Status and Student Achievement , 2000 .

[23]  Laura M. Desimone,et al.  Improving Impact Studies of Teachers’ Professional Development: Toward Better Conceptualizations and Measures , 2009 .

[24]  Harald Reiterer,et al.  Standards and software-ergonomic evaluation , 1995 .

[25]  Nigel Bevan,et al.  Human-Computer Interaction Standards , 1995 .

[26]  Huey-Tsyh Chen,et al.  A Comprehensive Typology for Program Evaluation , 1996 .

[27]  M. Patton.  Qualitative research and evaluation methods , 1980 .

[28]  Sandra Rollings-Magnusson,et al.  Legislation and Lifelong Learning in Canada: Inconsistencies in Implementation , 2001 .

[29]  Chris L. S. Coryn.  Using Hierarchical Linear Modeling for Proformative Evaluation: A Case Example , 2007, Journal of MultiDisciplinary Evaluation.

[30]  Gary T. Henry,et al.  Comparison Group Designs , 2015 .

[31]  S. Joy Mountford,et al.  The Art of Human-Computer Interface Design , 1990 .

[32]  Uwe Flick,et al.  Designing Qualitative Research , 2008 .

[33]  Susanna Loeb,et al.  The Draw of Home: How Teachers&Apos; Preferences for Proximity Disadvantage Urban Schools , 2003 .

[34]  T. Sass,et al.  Teacher training, teacher quality and student achievement , 2011 .

[35]  Hans G. Schuetze,et al.  Extending Access, Choice, and the Reign of the Market: Higher Education Reforms in British Columbia, 1989-2004 , 2004 .

[36]  S. Fagerhaugh,et al.  Participant Observation , 1979 .

[37]  Lyn Richards,et al.  Using NVIVO in Qualitative Research , 1999 .

[38]  David L. Weimer,et al.  Organizational report cards , 1999 .

[39]  Ron Thomas,et al.  Web-Based Instruction , 1997, Encyclopedia of Education and Information Technologies.

[40]  Diane J. Hanson,et al.  E-Learning: Strategies for Delivering Knowledge in the Digital Age , 2003, J. Educ. Technol. Soc..

[41]  Helen F. Ladd,et al.  How and Why Do Teacher Credentials Matter for Student Achievement? , 2007 .

[42]  Linda C. Martin.  Storyboarding Multimedia Interactions , 2000 .

[43]  Katherine Barber,et al.  Canadian Oxford Dictionary , 1998 .

[44]  James H. McMillan,et al.  Research in Education: A Conceptual Introduction , 1984 .

[45]  M. Scriven.  Types of Evaluation and Types of Evaluator , 1996 .

[46]  Robert M. Torres.  Lifelong Learning: A New Momentum and a New Opportunity for Adult Basic Learning and Education (ABLE) in the South. , 2004 .

[47]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[48]  Barry K. Beyer,et al.  How to conduct a formative evaluation , 1995 .

[49]  Thomas C. Reeves,et al.  Systematic evaluation procedures for interactive multimedia for education and training , 1996 .

[50]  Lorna Marsden.  Lifelong Learning: Crossing Generations and Cultures. , 2000 .

[51]  Dan Goldhaber,et al.  Can Teacher Quality Be Effectively Assessed? National Board Certification as a Signal of Effective Teaching , 2005, The Review of Economics and Statistics.

[52]  H. Russell Bernard,et al.  Social Research Methods: Qualitative and Quantitative Approaches , 2000 .

[53]  James O. Hamblen,et al.  Rapid Prototyping of Digital Systems , 1999 .

[54]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme , 1997 .

[55]  Richard Horn,et al.  Electronic performance support systems , 1997, CACM.

[56]  Laura M. Desimone,et al.  Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study , 2002 .

[57]  Eric A. Hanushek,et al.  Assessing the Effects of School Resources on Student Performance: An Update , 1997 .

[58]  Michael Hughes,et al.  Usability testing of Web-based training , 2001 .

[59]  Barbara N. Flagg.  Formative Evaluation for Educational Technologies , 1989 .

[60]  P. Twining.  Oversold and underused: computers in the classroom , 2002 .

[61]  Vivian C. Wong,et al.  Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within‐study comparisons , 2008 .

[62]  Steven D. Tripp,et al.  Rapid prototyping: An alternative instructional design strategy , 1990 .

[63]  Arend J. Visscher,et al.  Formative Evaluation in Educational Computing Research and Development , 1999 .

[64]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[65]  Peter M. Steiner,et al.  Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random and Nonrandom Assignments , 2008 .

[66]  Thomas C. Reeves,et al.  Usability testing and return-on-investment studies: key evaluation strategies for Web-based training , 2001 .

[67]  Gary T. Henry,et al.  Stayers and Leavers , 2011 .

[68]  Merle Conyer,et al.  User and Usability Testing--How It Should Be Undertaken?. , 1995 .

[69]  Gema Zamarro,et al.  Teacher qualifications and student achievement in urban elementary schools , 2009 .

[70]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[71]  Kathleen M. Gomoll,et al.  Some Techniques for Observing Users , 2001 .

[72]  Jonathan Stacks,et al.  Developmental Evaluation , 2011, Health promotion practice.

[73]  Harold W. Thimbleby,et al.  User interface design , 1990, ACM Press Frontier Series.

[74]  John Karat,et al.  Software Evaluation Methodologies , 1988 .

[75]  Craig A. Mertler,et al.  Introduction to Educational Research , 1988 .

[76]  Helen F. Ladd,et al.  Teacher Credentials and Student Achievement in High School: A Cross-Subject Analysis with Student Fixed Effects , 2007 .

[77]  David J. Flinders,et al.  Theory and Concepts in Qualitative Research: Perspectives from the Field , 1993 .

[78]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[79]  Hans G. Schuetze,et al.  Participation and exclusion: A comparative analysis of non-traditional students and lifelong learners in higher education , 2002 .

[80]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[81]  Candice Bowman,et al.  The role of formative evaluation in implementation research and the QUERI experience , 2006, Journal of General Internal Medicine.