Assessment of transparent and reproducible research practices in the psychiatry literature

Background Reproducibility is a cornerstone of scientific advancement; however, many published works may lack the core components needed for study reproducibility. Aims In this study, we evaluate the state of transparency and reproducibility in the field of psychiatry using specific indicators as proxies for these practices. Methods An increasing number of publications have investigated indicators of reproducibility, including research by Harwicke et al, from which we based the methodology for our observational, cross-sectional study. From a random 5-year sample of 300 publications in PubMed-indexed psychiatry journals, two researchers extracted data in a duplicate, blinded fashion using a piloted Google form. The publications were examined for indicators of reproducibility and transparency, which included availability of: materials, data, protocol, analysis script, open-access, conflict of interest, funding and online preregistration. Results This study ultimately evaluated 296 randomly-selected publications with a 3.20 median impact factor. Only 107 were available online. Most primary authors originated from USA, UK and the Netherlands. The top three publication types were cohort studies, surveys and clinical trials. Regarding indicators of reproducibility, 17 publications gave access to necessary materials, four provided in-depth protocol and one contained raw data required to reproduce the outcomes. One publication offered its analysis script on request; four provided a protocol availability statement. Only 107 publications were publicly available: 13 were registered in online repositories and four, ten and eight publications included their hypothesis, methods and analysis, respectively. Conflict of interest was addressed by 177 and reported by 31 publications. Of 185 publications with a funding statement, 153 publications were funded and 32 were unfunded. Conclusions Currently, Psychiatry research has significant potential to improve adherence to reproducibility and transparency practices. Thus, this study presents a reference point for the state of reproducibility and transparency in Psychiatry literature. Future assessments are recommended to evaluate and encourage progress.

[1]  Mallory C. Kidwell,et al.  An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017) , 2019, Royal Society Open Science.

[2]  B. Goldacre,et al.  COMPare: a prospective cohort study correcting and monitoring 58 misreported trials in real time , 2019, Trials.

[3]  M. Oremus,et al.  Selective outcome reporting is present in randomized controlled trials in lung cancer immunotherapies. , 2019, Journal of clinical epidemiology.

[4]  J. Ioannidis,et al.  Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017 , 2018, PLoS biology.

[5]  R. Sagar,et al.  Ethics in Psychiatric Research: Issues and Recommendations , 2017, Indian journal of psychological medicine.

[6]  Anna Falk Delgado,et al.  Outcome switching in randomized controlled oncology trials reporting on surrogate endpoints: a cross-sectional analysis , 2017, Scientific reports.

[7]  M. Vassar,et al.  Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals , 2017, PloS one.

[8]  Zhen Wang,et al.  Guidelines for reporting meta-epidemiological methodology research , 2017, Evidence-Based Medicine.

[9]  V. Bell.  Open science in mental health research. , 2017, The lancet. Psychiatry.

[10]  M. Vassar,et al.  Evidence of selective reporting bias in hematology journals: A systematic review , 2017, PloS one.

[11]  M. Vassar,et al.  Selective outcome reporting in obesity clinical trials: a cross‐sectional review , 2017, Clinical obesity.

[12]  S. Gandevia,et al.  Questionable science and reproducibility in electrical brain stimulation research , 2017, PloS one.

[13]  Christoph Werner,et al.  Trust in the health care professional and health outcome: A meta-analysis , 2017, PloS one.

[14]  Jennifer Urbano Blackford,et al.  Leveraging Statistical Methods to Improve Validity and Reproducibility of Research Findings , 2017, JAMA psychiatry.

[15]  R. Mulder,et al.  A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials , 2017, Acta psychiatrica Scandinavica.

[16]  M. Vassar,et al.  Shoulder Arthroplasty Trials Are Infrequently Registered: A Systematic Review of Trials , 2016, PloS one.

[17]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[18]  Benedikt V. Ehinger,et al.  Faculty Opinions recommendation of PSYCHOLOGY. Estimating the reproducibility of psychological science. , 2015 .

[19]  N. Christopherson,et al.  National Institutes of Health (NIH) , 2015 .

[20]  Michael C. Frank,et al.  Estimating the reproducibility of psychological science , 2015, Science.

[21]  R. Mulder,et al.  Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration , 2015, PloS one.

[22]  Brian A. Nosek,et al.  An open investigation of the reproducibility of cancer biology research , 2014, eLife.

[23]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions , 2010, International Coaching Psychology Review.

[24]  David Moher,et al.  Comparison of registered and published primary outcomes in randomized controlled trials. , 2009, JAMA.

[25]  R. Rosenthal,et al.  Selective publication of antidepressant trials and its influence on apparent efficacy. , 2008, The New England journal of medicine.

[26]  R. Horton,et al.  Clinical trial registration: looking back and moving ahead , 2007, Canadian Medical Association Journal.

[27]  Estimating the reproducibility of psychological science , 2015 .

[28]  G. Aghajanian,et al.  General psychiatry , 2008 .

[29]  IN REAL-TIME , 2008 .