Fighting Publication Bias: Introducing the Negative Results Section

Only data that are available via publications—and, to a certain extent, via presentations at conferences—can contribute to progress in the life sciences. However, it has long been known that a strong publication bias exists, in particular against the publication of data that do not reproduce previously published material or that refute the investigators’ initial hypothesis. The latter type of contradictory evidence is commonly known as ‘negative data.’ This slightly derogatory term reflects the bias against studies in which investigators were unable to reject their null hypothesis (H0), a tool of frequentist statistics that states that there is no difference between experimental groups. Researchers are well aware of this bias, as journals are usually not keen to publish the nonexistence of a phenomenon or treatment effect. They know that editors have little interest in publishing data that refute, or do not reproduce, previously published work—with the exception of spectacular cases that guarantee the attention of the scientific community, as well as garner extra citations (Ioannidis and Trikalinos, 2005). The authors of negative results are required to provide evidence for failure to reject the null hypothesis under numerous conditions (e.g., dosages, assays, outcome parameters, additional species or cell types), whereas a positive result would be considered worthwhile under any of these conditions (Rockwell et al, 2006). Indeed, there is a dilemma: one can never prove the absence of an effect, because, as Altman and Bland (1995) remind us, ‘absence of evidence is not evidence of absence’. It has been demonstrated that studies reporting positive, or significant, results are more likely to be published, and outcomes that are statistically significant have higher odds of being fully reported (Dwan et al, 2008). Negative results are more likely than positive results to be published in journals with lower impact factors (Littner et al, 2005). Many of you have experienced this phenomenon yourselves—often scientists mention in conversation that they ‘were not able to reproduce’ a particular finding, a statement that is very often countered by the question ‘Why did you not publish this? It would have been important for me to know.’ Publication bias has been systematically investigated, particularly in clinical trials (e.g., Liebeskind et al, 2006). Systematic reviews and meta-analyses have exposed the problem, as they are heavily confounded by this phenomenon (Sutton et al, 2000). Given a sufficiently large number of original studies, meta-analysis can even quantify the bias attributable to unpublished data. Where this has been done—for example, with Egger plots and trimand-fill analysis (Duval and Tweedie, 2000)—imputation of the probable results of the unpublished experiments not only reveals the amount of missing data but also estimates ‘true’ effect sizes resulting from the inclusion of the missing data. Quite commonly, a substantial proportion of the existing data appears to be missing. Inclusion of the modeled missing data into the meta-analysis sometimes results in a complete loss of the published effect of an intervention or the existence of a phenomenon. In many cases effect sizes shrink dramatically, hinting at the fact that very often the literature represents the ‘positive’ tip of an iceberg, whereas unpublished data loom below the surface. Such missing data would have the potential to have a significant impact on our pathophysiological understanding or treatment concepts. Only recently have systematic reviews been introduced into experimental medicine. Indeed, the stroke and cerebrovascular fields have pioneered this movement. These systematic reviews have exposed various sources of bias and produced the first indications that publication bias is highly prevalent (Macleod et al, 2004). Macleod and colleagues have now, for the first time, quantified publication bias in animal stroke studies and demonstrated that it leads to major overstatements of efficacy (Sena et al, 2010). The phenomenon of publication bias has long been known and long been bemoaned. Its substantial negative impact on science has been quantified. But how can we improve this lamentable situation, which may contribute greatly to our difficulties in translating bench findings to the bedside? The impetus must now come from the journals and publishers (De Maria, 2004; Diguet et al, 2004; Dirnagl, 2006; Knight, 2003). To our knowledge, only one journal in the neurosciences, Neurobiology of Aging, has thus far formally addressed the problem of negative Journal of Cerebral Blood Flow & Metabolism (2010) 30, 1263–1264 & 2010 ISCBFM All rights reserved 0271-678X/10 $32.00