The Certainty Framework for Assessing Real‐World Data in Studies of Medical Product Safety and Effectiveness

A fundamental question in using real‐world data for clinical and regulatory decision making is: How certain must we be that the algorithm used to capture an exposure, outcome, cohort‐defining characteristic, or confounder is what we intend it to be? We provide a practical framework to help researchers and regulators assess and classify the fit‐for‐purposefulness of real‐world data by study variable for a range of data sources. The three levels of certainty (optimal, sufficient, and probable) must be considered in the context of each study variable, the specific question being studied, the study design, and the decision at hand.

[1]  Sudha R. Raman,et al.  Linking electronic health data in pharmacoepidemiology: Appropriateness and feasibility , 2020, Pharmacoepidemiology and drug safety.

[2]  S. Hernández-Díaz,et al.  Using nationally representative survey data for external adjustment of unmeasured confounders: An example using the NHANES data , 2019, Pharmacoepidemiology and drug safety.

[3]  Sudha R. Raman,et al.  Data linkage in pharmacoepidemiology: A call for rigorous evaluation and reporting , 2019, Pharmacoepidemiology and drug safety.

[4]  N. Verlander,et al.  Influenza-associated mortality in hospital care: a retrospective cohort study of risk factors and impact of oseltamivir in an English teaching hospital, 2016 to 2017 , 2019, Euro surveillance : bulletin Europeen sur les maladies transmissibles = European communicable disease bulletin.

[5]  S. de Lusignan,et al.  ADVANCE system testing: Can coverage of pertussis vaccination be estimated in European countries using electronic healthcare databases: An example. , 2019, Vaccine.

[6]  K. Huybrechts,et al.  The International Society for Pharmacoepidemiology's Comments on the Core Recommendations in the Summary of the Heads of Medicines Agencies (HMA) ‐ EMA Joint Big Data Task Force , 2019, Pharmacoepidemiology and drug safety.

[7]  K. Ohe,et al.  Establishment of the MID‐NET® medical information database network as a reliable and valuable database for drug safety assessments in Japan , 2019, Pharmacoepidemiology and drug safety.

[8]  Robert B. Anderson,et al.  Lessons on Data Collection and Curation From the NFL Injury Surveillance Program , 2019, Sports health.

[9]  Nicolle M Gatto,et al.  A Structured Preapproval and Postapproval Comparative Study Design Framework to Generate Valid and Transparent Real‐World Evidence for Regulatory Decisions , 2019, Clinical pharmacology and therapeutics.

[10]  K. Haynes.  Mortality: The final outcome , 2019, Pharmacoepidemiology and drug safety.

[11]  R. Erichsen,et al.  Validating registry data on statins prescriptions by blood measurements , 2019, Pharmacoepidemiology and drug safety.

[12]  Alison Cave,et al.  Real‐World Data for Regulatory Decision Making: Challenges and Possible Solutions for Europe , 2019, Clinical pharmacology and therapeutics.

[13]  René De La Pedraja.  The Final Outcome , 2019, Energy Politics in Colombia.

[14]  David Martin,et al.  Evaluating the Use of Nonrandomized Real‐World Data Analyses for Regulatory Decision Making , 2019, Clinical pharmacology and therapeutics.

[15]  N. Dreyer,et al.  Considerations in characterizing real‐world data relevance and quality for regulatory purposes: A commentary , 2018, Pharmacoepidemiology and drug safety.

[16]  Olaf Klungel,et al.  The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE) , 2018, British Medical Journal.

[17]  C. Fuller,et al.  Overall and cause‐specific mortality in the Sentinel system: A power analysis , 2018, Pharmacoepidemiology and drug safety.

[18]  Keith Marsolo,et al.  Evaluating Foundational Data Quality in the National Patient-Centered Clinical Research Network (PCORnet®) , 2018, EGEMS.

[19]  Nancy A. Dreyer,et al.  Advancing a Framework for Regulatory Use of Real-World Evidence , 2018, Therapeutic innovation & regulatory science.

[20]  Keith Marsolo,et al.  A longitudinal analysis of data quality in a large pediatric data research network , 2017, J. Am. Medical Informatics Assoc..

[21]  David Madigan,et al.  Good practices for real‐world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR‐ISPE Special Task Force on real‐world evidence in health care decision making , 2017, Pharmacoepidemiology and drug safety.

[22]  Olaf Klungel,et al.  Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0 , 2017, Pharmacoepidemiology and drug safety.

[23]  Lisa M. LaVange,et al.  Multidimensional Evidence Generation and FDA Regulatory Decision Making: Defining and Using "Real-World" Data. , 2017, JAMA.

[24]  H. Sørensen,et al.  Data Resource Profile: The Danish National Prescription Registry , 2016, International journal of epidemiology.

[25]  N. Dreyer,et al.  The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness , 2016, Journal of managed care & specialty pharmacy.

[26]  James M Robins,et al.  Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available. , 2016, American journal of epidemiology.

[27]  Sigrun Alba Johannesdottir Schmidt,et al.  The Danish National Patient Registry: a review of content, data quality, and research potential , 2015, Clinical epidemiology.

[28]  M. Brookhart,et al.  Control Outcomes and Exposures for Improving Internal Validity of Nonrandomized Studies. , 2015, Health services research.

[29]  David B Richardson,et al.  The Active Comparator, New User Study Design in Pharmacoepidemiology: Historical Foundations and Contemporary Application , 2015, Current Epidemiology Reports.

[30]  R. Pazdur,et al.  FDA Approval: Blinatumomab , 2015, Clinical Cancer Research.

[31]  L. Buie,et al.  Blinatumomab , 2015, The Annals of pharmacotherapy.

[32]  Patrick B. Ryan,et al.  Transparent Reporting of Data Quality in Distributed Data Networks , 2015, EGEMS.

[33]  Richard F MacLehose,et al.  Good practices for quantitative bias analysis. , 2014, International journal of epidemiology.

[34]  Timothy L Lash,et al.  Comparison of Bias Analysis Strategies Applied to a Large Data Set , 2014, Epidemiology.

[35]  W Katherine Yih,et al.  Intussusception risk after rotavirus vaccination in U.S. infants. , 2014, The New England journal of medicine.

[36]  Takashi Ando,et al.  Claims-Based Definition of Death in Japanese Claims Database: Validity and Implications , 2013, PloS one.

[37]  P. Wahl,et al.  Validation of a claims‐based diagnostic code for Stevens–Johnson syndrome in a commercially insured population , 2012, Pharmacoepidemiology and drug safety.

[38]  Teresa To,et al.  Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. , 2011, Journal of clinical epidemiology.

[39]  J. Hallas,et al.  The Danish National Prescription Registry , 2011, Scandinavian journal of public health.

[40]  Rob Koeling,et al.  What does validation of cases in electronic record databases mean? The potential contribution of free text† , 2011, Pharmacoepidemiology and drug safety.

[41]  Sebastian Schneeweiss,et al.  Identification of hospitalizations for intentional self‐harm when E‐codes are incompletely recorded , 2010, Pharmacoepidemiology and drug safety.

[42]  M. Lipsitch,et al.  Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies , 2010, Epidemiology.

[43]  Robert T. Chen,et al.  The Brighton Collaboration: Creating a Global Standard for Case Definitions (and Guidelines) for Adverse Events Following Immunization , 2005 .

[44]  K. Rothman,et al.  Health outcomes among patients receiving oseltamivir , 2004, Pharmacoepidemiology and drug safety.

[45]  R. F. Allen.  A Commentary , 1966 .

[46]  Peter L. Elkin,et al.  ICD90CM Claims Data are Insufficient for Influenza Surveillance , 2013, MedInfo.

[47]  Guidance for Industry and FDA Staff Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data , 2013 .

[48]  Canary Wharf,et al.  The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) , 2012 .

[49]  S. Allison,et al.  Power analysis. , 2004, The Journal of orthopaedic and sports physical therapy.