Blinded by the Light: How a Focus on Statistical “Significance” May Cause p-Value Misreporting and an Excess of p-Values Just Below .05 in Communication Science

Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p < .05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below .05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below .05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p < .05 while in fact p > .05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below .05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.