Hidden Evidence Behind Useless Replications

Experiments that are run with few experimental subjects are often considered not to be very reliable and deemed, as a result, to be useless with a view to generating new knowledge. This belief is not, however, entirely correct. Today we have tools, such as metaanalysis, that we can use to aggregate small-scale experiments and output results that are equivalent to experiments run on large samples that are therefore reliable. The application of metaanalysis can overcome some of the obstacles that we come up against when running software engineering experiments (such as, for example, the practitioner availability problem).

[1]  Vigdis Kampenes,et al.  Quality of Design, Analysis and Reporting of Software Engineering Experiments:A Systematic Review , 2007 .

[2]  F. Miguez,et al.  Review of Corn Yield Response under Winter Cover Cropping Systems Using Meta-Analytic Methods , 2005 .

[3]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[4]  John A. Nelder,et al.  Generalized linear models for the analysis of quality‐improvement experiments , 1998 .

[5]  B. Becker,et al.  How meta-analysis increases statistical power. , 2003, Psychological methods.

[6]  F. Schmidt.  Meta-Analysis , 2008 .

[7]  James Miller,et al.  Applying meta-analytical procedures to software engineering experiments , 2000, J. Syst. Softw..

[8]  D. Moher,et al.  Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. , 2001, JAMA.

[9]  Larry V. Hedges,et al.  How hard is hard science, how soft is soft science? The empirical cumulativeness of research. , 1987 .

[10]  Forrest Shull,et al.  The empirical investigation of Perspective-Based Reading , 1995, Empirical Software Engineering.

[11]  L. Hedges,et al.  A Brief History of Research Synthesis , 2002, Evaluation & the health professions.

[12]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[13]  K. Arrow,et al.  The New Palgrave Dictionary of Economics , 2020 .

[14]  Jessica Gurevitch,et al.  Design and Analysis of Ecological Experiments , 1993 .

[15]  L. Hedges,et al.  The power of statistical tests in meta-analysis. , 2001, Psychological methods.

[16]  S. B. Thacker.  How Science Takes Stock: The Story of Meta-Analysis , 1998 .

[17]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research , 2008 .

[18]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.