Design Experiments: Engaging Policy Makers in the Search for Evidence about What Works

This article presents an argument for the greater use of design experiments, which can assist policy making because they provide both robust and timely evidence. We discuss their origins in education research, set out the methodology and propose some adaptations to the techniques used in these education studies to foster their application to a range of policy fields and problem areas. Design experiments need to meet two challenges. Can they provide valid evidence? Can they provide evidence that will be used by policy makers? Our argument shows how design experiments are robust when set against the classical canons of scientific study. We further claim that the design experiment approach offers a more viable means to developing evidence-based policy making than other forms of evaluation because of the timeliness of the insights that it provides.

[1]  Allan Collins,et al.  Design Research: Theoretical and Methodological Issues , 2004 .

[2]  Allan Collins,et al.  Toward a Design Science of Education , 1992 .

[3]  Ann L. Brown,et al.  Guided discovery in a community of learners. , 1994 .

[4]  Design-Based Research: An Emerging Paradigm for Educational Inquiry , 2003 .

[5]  L. Schauble,et al.  Design Experiments in Educational Research , 2003 .

[6]  Ann C. H. Kindfield,et al.  Integrating Curriculum, Instruction, Assessment, and Evaluation in a Technology-Supported Genetics Learning Environment , 2003 .

[7]  Peter Bryant,et al.  Design Experiments and Laboratory Approaches to Learning: Steps Toward Collaborative Exchange , 2003 .

[8]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[9]  E. Broady.  Finding out what works , 2004 .

[10]  Iris Tabak,et al.  BGuILE: Stragtegic and conceptual scaffolds for scientific inquiry in biology classrooms , 2001 .

[11]  I. Sanderson.  Evaluation, Policy Learning and Evidence-Based Policy Making , 2002 .

[12]  Abbie Brown,et al.  Design experiments: Theoretical and methodological challenges in creating complex interventions in c , 1992 .

[13]  A. diSessa.  Local sciences: viewing the design of human-computer systems as cognitive science , 1991 .

[14]  Annette Boaz,et al.  Social Science and the Evidence-based Policy Movement , 2002, Social Policy and Society.

[15]  R. Shavelson,et al.  Scientific Research in Education , 2002 .

[16]  Kurt Lewin,et al.  Resolving social conflicts : selected papers on group dynamics , 1948 .

[17]  Ernest T. Stringer.  Action research : a handbook for practitioners , 1996 .

[18]  Fadhel Kaboub.  Realistic Evaluation , 2004 .

[19]  David H. Greenberg,et al.  Social Experimentation and Public Policymaking , 2003 .

[20]  Philip Bell,et al.  On the Theoretical Breadth of Design-Based Research in Education , 2004 .

[21]  Robert J. McQueen,et al.  Can action research be made more rigorous in a positivist sense? The contribution of an iterative approach , 1997 .

[22]  H. Arendt.  Truth and Politics , 2008 .

[23]  Paul Horwitz,et al.  Hypermodels: Embedding Curriculum and Assessment in Computer-based Manipulatives. , 1999 .

[24]  Ann L. Brown,et al.  Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. , 1996 .

[25]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[26]  Diana Joseph.  The Practice of Design-Based Research: Uncovering the Interplay Between Design, Research, and the Real-World Context , 2004 .

[27]  Burt Perrin,et al.  How to — and How Not to — Evaluate Innovation , 2002 .

[28]  Paul Horwitz,et al.  Advancing Educational Theory by Enhancing Practice in a Technology-Supported Genetics Learning Environment. , 1999 .

[29]  Daniel C. Edelson.  Design Research: What We Learn When We Engage in Design , 2002 .

[30]  G. Stoker.  Why Politics Matters , 2016 .

[31]  H. B. English.  Resolving social conflicts. , 1949 .