Can Orthopaedics become the Gold Standard for Reproducibility? A Roadmap to Success

Background Scientific research is replete with poor accessibility to data, materials, and protocol, which limits the reproducibility of a study. Transparency with regard to materials, protocols, and raw data sets enhances reproducibility by providing the critical information necessary to verify, replicate, and resynthesize research findings. The extent to which transparency and reproducibility exist in the field of orthopaedics is unclear. In our study, we aimed to evaluate transparency and reproducibility-related characteristics of randomly sampled publications in orthopaedic journals. Methods We used the National Library of Medicine catalog to identify English language and MEDLINE-indexed orthopaedic journals. From the 74 journals meeting our inclusion criteria, we randomly sampled 300 publications using a refined PubMed search that were published between January 1, 2014, and December 31, 2018. Two investigators were trained for data extraction and analysis. Both investigators were blinded and independently extracted data from the 300 studies. Results Our initial search yielded 68,102 publications, from which we drew a random sample of 300 publications. Of these 300 publications, 286 were screened for empirical data and 14 were inaccessible. For analysis purposes, we excluded publications without empirical data. Of the 182 with empirical data, 13 studies (7.1%) included a data availability statement, 9 (4.9%) reported materials were available, none (0.0%) provided analysis scripts, 2 (1.1%) provided access to the protocol used, 5 (2.7%) were preregistered, and only 2 (1.1%) provided a statement about being a replicated study. Conclusions Components necessary for reproducibility are lacking in orthopaedic surgery journals. The vast majority of publications did not provide data or material availability statements, protocols, or analysis scripts, and had no preregistration statements. Intervention is needed to improve reproducibility in the field of orthopaedics. The current state of reproducibility in orthopaedic surgery could be improved by combined efforts from funding agencies, authors, peer reviewers, and journals alike. Level of Evidence N/A

[1]  John P. A. Ioannidis,et al.  Research: increasing value, reducing waste 2 , 2014 .

[2]  J. Kaiser.  Plan to replicate 50 high-impact cancer papers shrinks to just 18 , 2018, Science.

[3]  Michèle B. Nuijten,et al.  Journal Data Sharing Policies and Statistical Reporting Inconsistencies in Psychology , 2017 .

[4]  Mallory C. Kidwell,et al.  An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017) , 2019, Royal Society Open Science.

[5]  M. Vassar,et al.  An Evaluation of Publication Bias in High-Impact Orthopaedic Literature , 2019, JB & JS open access.

[6]  John P. A. Ioannidis,et al.  Reproducible Research Practices and Transparency across the Biomedical Literature , 2016, PLoS biology.

[7]  Elizabeth Gilbert,et al.  Reproducibility Project: Results (Part of symposium called "The Reproducibility Project: Estimating the Reproducibility of Psychological Science") , 2014 .

[8]  M. Vassar,et al.  An Evaluation of Spin in Lower Extremity Joint Trials. , 2019, The Journal of arthroplasty.

[9]  D. Burton,et al.  Arthroscopic subacromial decompression for subacromial shoulder pain (CSAW): a multicentre, pragmatic, parallel group, placebo-controlled, three-group, randomised surgical trial , 2017, The Lancet.

[10]  Brian A. Nosek,et al.  Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 , 2018, Nature Human Behaviour.

[11]  John P. A. Ioannidis,et al.  What does research reproducibility mean? , 2016, Science Translational Medicine.

[12]  Michael C. Frank,et al.  A practical guide for transparency in psychological science , 2018 .

[13]  J. Ioannidis,et al.  Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017 , 2018, PLoS biology.

[14]  P. Glasziou,et al.  Avoidable waste in the production and reporting of research evidence , 2009, The Lancet.

[15]  R. Tibshirani,et al.  Increasing value and reducing waste in research design, conduct, and analysis , 2014, The Lancet.

[16]  Michael C. Frank,et al.  Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition , 2018, Royal Society Open Science.

[17]  M. Baker.  1,500 scientists lift the lid on reproducibility , 2016, Nature.

[18]  M. Vassar,et al.  An Evaluation of Reporting Guidelines and Clinical Trial Registry Requirements Among Orthopaedic Surgery Journals , 2018, The Journal of bone and joint surgery. American volume.