The misdirection of public policy: comparing and combining standardised effect sizes

Abstract Increased attention on ‘what works’ in education has led to an emphasis on developing policy from evidence based on comparing and combining a particular statistical summary of intervention studies: the standardised effect size. It is assumed that this statistical summary provides an estimate of the educational impact of interventions and combining these through meta-analyses and meta-meta-analyses results in more precise estimates of this impact which can then be ranked. From these, it is claimed, educational policy decisions can be driven. This paper will demonstrate that these assumptions are false: standardised effect size is open to researcher manipulations which violate the assumptions required for legitimately comparing and combining studies in all but the most restricted circumstances. League tables of types of intervention, which governments point to as an evidence base for effective practice may, instead, be hierarchies of openness to research design manipulations. The paper concludes that public policy and resources are in danger of being misdirected.

[1]  Andrea B. Bermudez,et al.  Using Peer Response Groups with Limited English Proficient Writers , 1993 .

[2]  K. Sylva,et al.  Influences on students ’ GCSE attainment and progress at age 16 Effective Pre-School , Primary & Secondary Education Project ( EPPSE ) Research Brief September 2014 , 2014 .

[3]  H. Walberg,et al.  The Effects of Homework on Learning: A Quantitative Synthesis , 1984 .

[4]  Maria Cardelle-Elawar,et al.  Effects of Teaching Metacognitive Skills to Students with Low Mathematics Ability. , 1992 .

[5]  Carol Rolheiser,et al.  Effects of Self-Evaluation Training on Narrative Writing. , 1999 .

[6]  M. O. Oladunni.  An experimental study on the effectiveness of metacognitive and heuristic problem solving techniques on computational performance of students in mathematics , 1998 .

[7]  Nancy Cartwright,et al.  Making the most of the evidence : evidence-based policy in the classroom. , 2015 .

[8]  R. Butler.  Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. , 1987 .

[9]  G. Camilli,et al.  Meta-Analysis of the Effects of Early Education Interventions on Cognitive and Social Development , 2010, Teachers College Record: The Voice of Scholarship in Education.

[10]  Charles R. Crowell,et al.  Performance posting, goal setting, and activity-contingent praise as applied to a university hockey team. , 1988 .

[11]  M. S. Patel,et al.  An introduction to meta-analysis. , 1989, Health Policy.

[12]  Harris Cooper,et al.  Does Homework Improve Academic Achievement? A Synthesis of Research, 1987–2003 , 2006 .

[13]  Hans J. Eysenck,et al.  Meta-Analysis: an Abuse of Research Integration , 1984 .

[14]  Robert J. Marzano,et al.  A Theory-Based Meta-Analysis of Research on Instruction. , 1998 .

[15]  Mark Asbridge,et al.  A meta-analysis of the effects of texting on driving. , 2014, Accident; analysis and prevention.

[16]  Eileen Kintsch,et al.  Summary Street: Interactive Computer Support for Writing , 2004 .

[17]  C. Fitz-Gibbon.  Meta‐analysis: an explication , 1984 .

[18]  L. Bickman,et al.  The SAGE Handbook of Applied Social Research Methods , 2009 .

[19]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[20]  Peter Henderson,et al.  The Sutton Trust - Education Endowment Foundation Teaching and Learning Toolkit. , 2013 .

[21]  Chen-Lin C. Kulik,et al.  The Instructional Effect of Feedback in Test-Like Events , 1991 .

[22]  W. Overton,et al.  Area and volume conservation among the elderly: Assessment and training. , 1976 .

[23]  L. Lumbelli,et al.  Improving the ability to detect comprehension problems: from revising to writing , 1999 .

[24]  Joe Dan Austin,et al.  Homework Grading Procedures in Junior High Mathematics Classes , 1974 .

[25]  F Dan Richard,et al.  Meta-analysis of raw mean differences. , 2003, Psychological methods.

[26]  M. Lacher.  Effects of Feedback, Instruction, and Initial Performance level upon Training and Persistence of Verbal Rehearsal. , 1983, The Journal of general psychology.

[27]  Sharon Vaughn,et al.  How effective are one-to-one tutoring programs in reading for elementary students at risk for reading failure? A meta-analysis of the intervention research , 2000 .

[28]  Jacob Cohen,et al.  The statistical power of abnormal-social psychological research: a review. , 1962, Journal of abnormal and social psychology.

[29]  Vivienne Baumfield,et al.  A meta-analysis of the impact of the implementation of thinking skills approaches on pupils. , 2005 .

[30]  Chun-Yen Chang,et al.  The use of a problem-solving-based instructional model in initiating change in students' achievement and alternative frameworks , 1999 .

[31]  Jim Garrison,et al.  Philosophy of Education , 1936 .

[32]  P. Lancaster,et al.  A meta-analysis of effects of , 2014 .

[33]  P. Davies.  What is Evidence-based Education? , 1999 .

[34]  Jane M. Singh.  Research in Homework as the Motivating Factor in Reading Achievement , 1970 .

[35]  Nancy Cartwright,et al.  Evidence-Based Policy: A Practical Guide to Doing It Better , 2012 .

[36]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[37]  Roland F. Gray,et al.  An Experimental Study of the Relationship of Homework to Pupil Success in Computation with Fractions , 1971 .

[38]  Terence R. Mitchell,et al.  Effects of assigned versus participatively set goals, knowledge of results, and individual differences on employee behavior when goal difficulty is held constant. , 1979 .

[39]  R. P. Carver.  The Case Against Statistical Significance Testing , 1978 .

[40]  Steve Graham,et al.  Formative Assessment and Writing , 2015, The Elementary School Journal.

[41]  A. Kluger,et al.  The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. , 1996 .

[42]  Robert M. Bernard,et al.  Instructional Interventions Affecting Critical Thinking Skills and Dispositions: A Stage 1 Meta-Analysis , 2008 .

[43]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[44]  Geoffrey Nelson,et al.  A Meta-Analysis of Longitudinal Research on Preschool Prevention Programs for Children. , 2003 .

[45]  Philip C. Abrami,et al.  Improving literacy and metacognition with electronic portfolios: Teaching and learning with ePEARL , 2010, Comput. Educ..

[46]  D. W. Schumann,et al.  Effects of corrective testing: an extension , 1984 .

[47]  L. Epstein,et al.  Improving Oral Hygiene with Videotape Modeling , 1981 .

[48]  G. Stuart.  Underachievement in education by white working class children : Government response to the Committee's first report of session 2014-15 second special report of session 2014-15 , 2014 .

[49]  Laurie M. Anderson,et al.  The effectiveness of early childhood development programs. A systematic review. , 2003, American journal of preventive medicine.

[50]  J. A. Lewis-Beck.  Locus of Control, Task Expectancies, and Children’s Performance following Failure , 1978 .

[51]  Lynn S. Fuchs,et al.  The Contribution of Skills Analysis to Curriculum-Based Measurement in Spelling , 1991 .

[52]  Neal M. Kingston,et al.  Formative Assessment: A Meta-Analysis and a Call for Research , 2011 .

[53]  D. Routh,et al.  Response cost, reinforcement, and children's porteus maze qualitative performance , 1986, Journal of abnormal child psychology.

[54]  Qing Li,et al.  A Meta-analysis of the Effects of Computer Technology on School Students’ Mathematics Learning , 2010 .