Standards and guidelines for observational studies: quality is in the eye of the beholder.

OBJECTIVES Patient care decisions demand high-quality research. To assist those decisions, numerous observational studies are being performed. Are the standards and guidelines to assess observational studies consistent and actionable? What policy considerations should be considered to ensure decision makers can determine if an observational study is of high-quality and valid to inform treatment decisions? STUDY DESIGN AND SETTING Based on a literature review and input from six experts, we compared and contrasted nine standards/guidelines using 23 methodological elements involved in observational studies (e.g., study protocol, data analysis, and so forth). RESULTS Fourteen elements (61%) were addressed by at least seven standards/guidelines; 12 of these elements disagreed in the approach. Nine elements (39%) were addressed by six or fewer standards/guidelines. Ten elements (43%) were not actionable in at least one standard/guideline that addressed the element. CONCLUSION The lack of observational study standard/guideline agreement may contribute to variation in study conduct; disparities in what is considered credible research; and ultimately, what evidence is adopted. A common set of agreed on standards/guidelines for conducting observational studies will benefit funders, researchers, journal editors, and decision makers.

[1]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[2]  Scott R. Smith,et al.  Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide , 2013 .

[3]  John B. Watkins,et al.  Standardizing Quality Assessment of Observational Studies for Decision Making in Health Care , 2009, Journal of managed care pharmacy : JMCP.

[4]  N. Dreyer,et al.  The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution , 2014, Journal of managed care & specialty pharmacy.

[5]  M. Halpern,et al.  Pharmaceutical Technology Assessment: Perspectives from Payers , 2012, Journal of managed care pharmacy : JMCP.

[6]  W. Sauerbrei,et al.  STRengthening Analytical Thinking for Observational Studies: the STRATOS initiative , 2014, Statistics in medicine.

[7]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[8]  Nicholas G. Hall,et al.  A Model for Making Project Funding Decisions at the National Cancer Institute , 1992, Oper. Res..

[9]  David Atkins,et al.  Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part I. , 2009, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[10]  Michael L. Johnson,et al.  Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report--Part III. , 2009, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[11]  Rachael Fleurence,et al.  How the Patient-Centered Outcomes Research Institute is engaging patients and others in shaping its research agenda. , 2013, Health affairs.

[12]  D. Meltzer,et al.  Toward a Science of Research Prioritization? The Use of Value of Information by Multidisciplinary Stakeholder Groups , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[13]  Uwe Siebert,et al.  Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retr , 2009, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[14]  A. Holtorf,et al.  Current and Future Use of HEOR Data in Healthcare Decision-Making in the United States and in Emerging Markets. , 2012, American health & drug benefits.

[15]  H. A. Lingstone,et al.  The Delphi Method: Techniques and Applications , 1976 .

[16]  C Daniel Mullins,et al.  A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. , 2014, Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research.

[17]  Richard E Gliklich,et al.  GRACE principles: recognizing high-quality observational studies of comparative effectiveness. , 2010, The American journal of managed care.

[18]  R. Brook,et al.  Consensus methods: characteristics and guidelines for use. , 1984, American journal of public health.