Statistically significant papers in psychiatry were cited more often than others.

OBJECTIVE Citations by other researchers are important in the dissemination of research findings. We aimed to investigate whether preferential citation of statistically significant articles exists in the psychiatric literature. STUDY DESIGN AND SETTINGS We analyzed all original research papers published in 1996 in four psychiatric journals. Using a standardized questionnaire, from each paper, we extracted the primary outcome and its statistical significance. The number of citations, excluding authors' "self-citations," received by April 2005 was obtained. Regression analysis was used to relate citation frequency to statistical significance, adjusting for confounders. RESULTS Of 448 extracted papers, 368 used statistical significance testing and 287 (77.8%) reported P<0.05. The median number of citations for papers reporting "significant" and "nonsignificant" results was 33 vs. 16. After adjustment for journal, study design, reporting quality, whether outcome confirmed previous findings and study size, the ratio of the number of citations per article for articles reporting "P<0.05" on the primary outcome to those reporting "P>0.05" was 1.63 (95% confidence interval [CI]: 1.32, 2.02, P<0.001). CONCLUSION Authors cite studies based on their P-value rather than intrinsic scientific merit. This practice skews the research evidence. Systematic study registration and inclusion in meta-analysis should be encouraged.

[1]  D. Altman.  Poor-quality medical research: what can journals do? , 2002, JAMA.

[2]  P. McCullagh,et al.  Generalized Linear Models , 1992 .

[3]  Jean Tague-Sutcliffe,et al.  An Introduction to Informetrics , 1992, Inf. Process. Manag..

[4]  A. Oxman,et al.  Comprehensiveness and bias in reporting clinical trials. Study of reviews of pneumococcal vaccine effectiveness. , 1995, Canadian family physician Medecin de famille canadien.

[5]  Herbert Snyder,et al.  Patterns of self-citation across disciplines (1980-1989) , 1998, J. Inf. Sci..

[6]  Pentti Nieminen,et al.  The effect of statistical methods and study reporting characteristics on the number of citations: A study of four general psychiatric journals , 2003, Scientometrics.

[7]  D G Altman,et al.  Statistics in medical journals: developments in the 1980s. , 1991, Statistics in medicine.

[8]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[9]  P. Gøtzsche.  Reference bias in reports of drug trials. , 1987, British medical journal.

[10]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[11]  Thomas A Trikalinos,et al.  Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. , 2005, Journal of clinical epidemiology.

[12]  H. Moed.  Citation Analysis in Research Evaluation (Information Science & Knowledge Management) , 2005 .

[13]  J. Miettunen,et al.  Statistical methodology in general psychiatric journals , 2002, Nordic journal of psychiatry.

[14]  L. Beach,et al.  The citation bias: Fad and fashion in the judgment and decision literature. , 1984 .

[15]  J. Pantel,et al.  [The evaluation of research performance in psychiatry. Potential and pitfalls of bibliometric analysis]. , 1999, Der Nervenarzt.

[16]  Eugene Garfield,et al.  Citation indexing - its theory and application in science, technology, and humanities , 1979 .

[17]  Leo Egghe,et al.  Introduction to Informetrics: Quantitative Methods in Library, Documentation and Information Science , 1990 .

[18]  R. Wears,et al.  Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. , 2002, JAMA.

[19]  John P A Ioannidis,et al.  Genetic associations: false or true? , 2003, Trends in molecular medicine.

[20]  Theodor D. Sterling,et al.  Publication decisions revisited: the effect of the outcome of statistical tests on the decision to p , 1995 .

[21]  Henk F. Moed,et al.  Citation Analysis in Research Evaluation , 1899 .

[22]  P. Nieminen,et al.  Type of empirical research reports as an explanatory factor in citation performance of psychiatric research , 1996, Scientometrics.

[23]  Wolfgang Glänzel,et al.  A bibliometric approach to the role of author self-citations in scientific communication , 2004, Scientometrics.

[24]  S. Golder,et al.  The effectiveness and cost-effectiveness of prophylactic removal of wisdom teeth. , 2000, Health technology assessment.

[25]  J. Ioannidis.  Contradicted and initially stronger effects in highly cited clinical research. , 2005, JAMA.

[26]  Jean-Christophe Doré,et al.  Citation bias in medical journals , 1999, Scientometrics.

[27]  C. Gluud,et al.  Citation bias of hepato-biliary randomized clinical trials. , 2002, Journal of clinical epidemiology.

[28]  G Koren,et al.  Bias against negative studies in newspaper reports of medical research. , 1991, JAMA.

[29]  Alex J. Sutton,et al.  Publication and related biases: a review , 2000 .

[30]  K. Dickersin.  The existence of publication bias and risk factors for its occurrence. , 1990, JAMA.

[31]  G. Wilkinson,et al.  Impact factors of psychiatric journals , 1997, British Journal of Psychiatry.

[32]  C. Begg,et al.  Publication bias : a problem in interpreting medical data , 1988 .

[33]  A. Carter,et al.  A survey identified publication bias in the secondary literature. , 2006, Journal of clinical epidemiology.

[34]  A. Hrõbjartsson,et al.  Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. , 2004, JAMA.

[35]  Wolfgang Glänzel,et al.  Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies , 2004, Scientometrics.

[36]  U. Ravnskov,et al.  Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. , 1992, BMJ.

[37]  W. Shadish,et al.  Author Judgements about Works They Cite: Three Studies from Psychology Journals , 1995 .

[38]  P. Seglen,et al.  Citations and journal impact factors: questionable indicators of research quality , 1997, Allergy.

[39]  L. Gluud.  Bias in clinical intervention research. , 2006, American journal of epidemiology.