Peer Review and Transparency in Evidence-Source Selection in Value and Health Technology Assessment.

OBJECTIVES Value and health technology assessment (V/HTA) is often used in clinical, access, and reimbursement decisions. V/HTA data-source selection may not be transparent, which is a necessary element for stakeholder understanding and trust and for fostering accountability among decision makers. Peer review is considered one mechanism for judging data trustworthiness. Our objective was (1) to use publicly available documentation of V/HTA methods to identify requirements for inclusion of peer-reviewed evidence sources, (2) to compare and contrast US and non-US approaches, and (3) to assess evidence sources used in published V/HTA reports. METHODS Publicly available methods documentation from 11 V/HTA organizations in North America and Europe were manually searched and abstracted for descriptions of requirements and recommendations regarding search strategy and evidence-source selection. The bibliographies of a subset of V/HTA reports published in 2018 were manually abstracted for evidence-source types used in each. RESULTS Heterogeneity in evidence-source retrieval and selection was observed across all V/HTA organizations, with more pronounced differences between US and non-US organizations. Not all documentation of organizations' methods address the evidence-source selection processes (7 of 11), and few explicitly reference peer-reviewed sources (3 of 11). Documentation of the evidence-source selection strategy was inconsistent across reports (6 of 13), and the level of detail provided varied across organizations. Some information on evidence-source selection was often included in confidential documentation and was not publicly available. CONCLUSIONS Disparities exist among V/HTA organizations in requirements and guidance regarding evidence-source selection. Standardization of evidence-source selection strategies and documentation could help improve V/HTA transparency and has implications for decision making based on report findings.

[1]  F. Collins,et al.  Policy: NIH plans to enhance reproducibility , 2014, Nature.

[2]  Maarten J. IJzerman,et al.  Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. , 2019, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[3]  A. DeMaria Peer review: better than the alternatives. , 2002, Journal of the American College of Cardiology.

[4]  Athina Tatsioni,et al.  Who is afraid of reviewers’ comments? Or, why anything can be published and anything can be cited , 2010, European journal of clinical investigation.

[5]  Jennifer M. Urban,et al.  Shining Light into Black Boxes , 2012, Science.

[6]  J. Sharfstein,et al.  Enhancing Transparency at the US Food and Drug Administration: Moving Beyond the 21st Century Cures Act. , 2017, JAMA.

[7]  S. Goodman,et al.  Reproducible Research: Moving toward Research the Public Can Really Trust , 2007, Annals of Internal Medicine.

[8]  Aris Angelis,et al.  Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight European countries , 2017, The European Journal of Health Economics.

[9]  J. Denis,et al.  Use of health technology assessment in decision making: Coresponsibility of users and producers? , 2005, International Journal of Technology Assessment in Health Care.

[10]  H. Kölsch,et al.  Reporting bias in medical research - a narrative review , 2010, Trials.

[11]  Harpreet S. Kohli,et al.  Health Technology Assessment and Health Policy-Making in Europe – Current Status, Challenges and Potential , 2010 .

[12]  Carol Tenopir,et al.  Trustworthiness and authority of scholarly information in a digital age: Results of an international questionnaire , 2016, J. Assoc. Inf. Sci. Technol..

[13]  Guido Rasi,et al.  Transparency and the European Medicines Agency--sharing of clinical trial data. , 2014, The New England journal of medicine.

[14]  S. Sullivan,et al.  Economic Evaluation in the US , 2012, PharmacoEconomics.

[15]  Louis P Garrison,et al.  A Health Economics Approach to US Value Assessment Frameworks-Introduction: An ISPOR Special Task Force Report [1]. , 2018, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[16]  Olaf Klungel,et al.  Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0 , 2017, Pharmacoepidemiology and drug safety.

[17]  Richard Smith,et al.  Peer Review: A Flawed Process at the Heart of Science and Journals , 2006, Journal of the Royal Society of Medicine.

[18]  William Hollingworth,et al.  Model Transparency and Validation , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[19]  R. Spier The history of the peer-review process. , 2002, Trends in biotechnology.

[20]  J. Ioannidis,et al.  Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017 , 2018, PLoS biology.

[21]  Scott D Ramsey,et al.  Health technology assessment in health-care decisions in the United States. , 2009, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.