Challenges in Building Usable Knowledge in Education

ABSTRACT The scientific rigor of education research has improved dramatically since the year 2000. Much of the credit for this improvement is deserved by Institute of Education Sciences (IES) policies that helped create a demand for rigorous research; increased human capital capacity to carry out such work; provided funding for the work itself; and collected, evaluated, and made available the results of that work through the What Works Clearinghouse. Major challenges still remain for education research, however. One challenge is dealing with the replication crisis that has plagued other scientific fields and is likely to be a problem in education science. A second challenge is better supporting the generalizability of education research. A third challenge is adapting our rigorous research designs to the increasing complexity of our interventions and our questions about the mechanisms by which these interventions achieve their effects. Promising approaches to meet each of these challenges are suggested.

[1]  L. Hedges,et al.  Estimation of Population Average Treatment Effects in the FIRST Trial: Application of a Propensity Score-Based Stratification Approach. , 2018, Health services research.

[2]  Elizabeth Tipton,et al.  Implications of Small Samples for Generalization: Adjustments and Rules of Thumb , 2017, Evaluation review.

[3]  Jeanette W. Chung,et al.  National Cluster-Randomized Trial of Duty-Hour Flexibility in Surgical Training , 2016 .

[4]  S. Hewitt,et al.  Reproducibility , 2019, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[5]  Gideon Nave,et al.  Evaluating replicability of laboratory experiments in economics , 2016, Science.

[6]  Inbal Nahum-Shani,et al.  Treatment Sequencing for Childhood ADHD: A Multiple-Randomization Study of Adaptive Medication and Behavioral Interventions , 2016, Journal of clinical child and adolescent psychology : the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53.

[7]  K. Fiedler,et al.  Questionable Research Practices Revisited , 2016 .

[8]  R. Kaplan,et al.  Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time , 2015, PloS one.

[9]  David R. Olson,et al.  Jerome Bruner: The Cognitive Revolution in Educational Theory , 2014 .

[10]  Robert E. Ployhart,et al.  Hierarchical Linear Models , 2014 .

[11]  B. Spring,et al.  Optimization of remotely delivered intensive lifestyle treatment for obesity using the Multiphase Optimization Strategy: Opt-IN study protocol. , 2014, Contemporary clinical trials.

[12]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[13]  S. Perrin Preclinical research: Make mouse studies work , 2014, Nature.

[14]  L. Hedges,et al.  Generalizing from unrepresentative experiments: a stratified propensity score approach , 2014 .

[15]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[16]  Leif D. Nelson,et al.  P-Curve: A Key to the File Drawer , 2013, Journal of experimental psychology. General.

[17]  F. Collins,et al.  NIH plans to enhance reproducibility , 2014 .

[18]  Elizabeth Tipton Improving Generalizations From Experiments Using Propensity Score Subclassification , 2013 .

[19]  Gregory Schraw,et al.  On Going (Way) Beyond One’s Data: A Proposal to Restrict Recommendations for Practice in Primary Educational Research Journals , 2013 .

[20]  Oscar Lovell Triggs,et al.  The Changing Order , 2013 .

[21]  D. Rennie,et al.  SPIRIT 2013 statement: defining standard protocol items for clinical trials. , 2013, Annals of internal medicine.

[22]  David Moher,et al.  SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials , 2013, BMJ.

[23]  L. Blanton,et al.  Teacher Education Reform Initiatives and Special Education: Convergence, Divergence, and Missed Opportunities , 2013 .

[24]  Stephen H Bell,et al.  External Validity in Policy Evaluations that Choose Sites Purposively. , 2013, Journal of policy analysis and management : [the journal of the Association for Public Policy Analysis and Management].

[25]  H. Pashler,et al.  Is the Replicability Crisis Overblown? Three Arguments Examined , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[26]  G. Loewenstein,et al.  Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling , 2012, Psychological science.

[27]  F. Prinz,et al.  Believe it or not: how much can we rely on published data on potential drug targets? , 2011, Nature Reviews Drug Discovery.

[28]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[29]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[30]  Camilla A. Heid,et al.  Head Start Impact Study. Final Report. , 2010 .

[31]  Louis M. Gomez,et al.  Getting Ideas into Action: Building Networked Improvement Communities in Education , 2010 .

[32]  Xiao-Hua Zhou,et al.  Statistical Methods for Meta‐Analysis , 2008 .

[33]  S. Murphy,et al.  The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. , 2007, American journal of preventive medicine.

[34]  N. Steneck ORI Introduction to Responsible Conduct of Research , 2007 .

[35]  Pamela A. Moss,et al.  Standards for Reporting on Empirical Social Science Research in AERA Publications American Educational Research Association , 2006 .

[36]  B. Schneider,et al.  Standards for Reporting on Empirical Social Science Research in AERA Publications American Educational Research Association , 2006 .

[37]  J. Ioannidis Why Most Published Research Findings Are False , 2005, PLoS medicine.

[38]  Vijay N. Nair,et al.  A strategy for optimizing and evaluating behavioral interventions , 2005, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[39]  J. Ioannidis Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[40]  J. Gill Hierarchical Linear Models , 2005 .

[41]  G. Miller The cognitive revolution: a historical perspective , 2003, Trends in Cognitive Sciences.

[42]  Y. Lincoln,et al.  Scientific Research in Education , 2004 .

[43]  Catherine P. Bradshaw,et al.  The use of propensity scores to assess the generalizability of results from randomized trials , 2011, Journal of the Royal Statistical Society. Series A,.

[44]  Larry V. Hedges,et al.  The Effects of Small Classes on Academic Achievement: The Results of the Tennessee Class Size Experiment , 2000 .

[45]  A. Schirm,et al.  The Impacts for Upward Bound: Final Report for Phase I of the National Evaluation , 1999 .

[46]  D Moher,et al.  CONSORT: an evolving tool to help improve the quality of reports of randomized controlled trials. Consolidated Standards of Reporting Trials. , 1998, JAMA.

[47]  J. Heckman,et al.  Understanding the Role of Cognitive Ability in Accounting for the Recent Rise in the Economic Return to Education , 1998 .

[48]  Hayes,et al.  Review of particle properties. , 1978, Physical review. D, Particles and fields.

[49]  Carl F. Kaestle,et al.  Research News and comment: The Awful Reputation of Education Research , 1993 .

[50]  N. Gage,et al.  The Paradigm Wars and Their Aftermath A “Historical” Sketch of Research on Teaching Since 1989 , 1989 .

[51]  Larry V. Hedges,et al.  How hard is hard science, how soft is soft science? The empirical cumulativeness of research. , 1987 .

[52]  Larry V. Hedges,et al.  Estimation of Effect Size under Nonrandom Sampling: The Effects of Censoring Studies Yielding Statistically Insignificant Mean Differences , 1984 .

[53]  L. Cronbach,et al.  Designing evaluations of educational and social programs , 1983 .

[54]  L. Cronbach Beyond the Two Disciplines of Scientific Psychology. , 1975 .

[55]  A. H. Rosenfeld,et al.  The Particle Data Group: Growth and Operations-Eighteen Years of Particle Physics , 1975 .