Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection

AbstractBackground Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. Objectives To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using “positive” and “negative” drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. Methods Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. Results The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45–73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82–96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78–91 %) and accuracy (78–72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. Conclusion Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.

[1]  S. Nissen,et al.  Rosiglitazone revisited: an updated meta-analysis of risk for myocardial infarction and cardiovascular mortality. , 2010, Archives of internal medicine.

[2]  Chieko Ishiguro,et al.  Analysis of the Factors Influencing the Spontaneous Reporting Frequency of Drug Safety Issues Addressed in the FDA’s Drug Safety Communications, Using FAERS Data , 2014, Pharmaceutical Medicine.

[3]  Manfred Hauben,et al.  ‘Extreme Duplication’ in the US FDA Adverse Events Reporting System Database , 2007, Drug safety.

[4]  Jatinder Singh.  International conference on harmonization of technical requirements for registration of pharmaceuticals for human use , 2015, Journal of pharmacology & pharmacotherapeutics.

[5]  A. Bate,et al.  Data-mining analyses of pharmacovigilance signals in relation to relevant comparison drugs , 2002, European Journal of Clinical Pharmacology.

[6]  David Madigan,et al.  Disproportionality methods for pharmacovigilance in longitudinal observational databases , 2013, Statistical methods in medical research.

[7]  Anika Ashok,et al.  Guidance for Industry by U.S. Department of Health and Human Services—Food and Drug Administration—Center for Biologics Evaluation and Research (CBER)—February 1999 , 2009 .

[8]  Martijn J. Schuemie,et al.  A Reference Standard for Evaluation of Methods for Drug Safety Signal Detection Using Electronic Healthcare Record Databases , 2012, Drug Safety.

[9]  Marius Fieschi,et al.  Harmonization process for the identification of medical events in eight European healthcare databases: the experience from the EU-ADR project , 2013, J. Am. Medical Informatics Assoc..

[10]  Judith A Racoosin,et al.  Utilizing Medicare claims data for real‐time drug safety evaluations: is it feasible? , , 2011, Pharmacoepidemiology and drug safety.

[11]  D. Madigan,et al.  Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership , 2012, Statistics in medicine.

[12]  Martijn J Schuemie,et al.  Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARD , 2011, Pharmacoepidemiology and drug safety.

[13]  G. D. Dal Pan,et al.  Estimating the extent of reporting to FDA: a case study of statin‐associated rhabdomyolysis , 2008, Pharmacoepidemiology and drug safety.

[14]  Rita Ouellet-Hellstrom,et al.  Risk of acute myocardial infarction, stroke, heart failure, and death in elderly Medicare patients treated with rosiglitazone or pioglitazone. , 2010, JAMA.

[15]  Jeffrey R Curtis,et al.  Adaptation of Bayesian Data Mining Algorithms to Longitudinal Claims Data: Coxib Safety as an Example , 2008, Medical care.

[16]  I. Edwards,et al.  Statins, Neuromuscular Degenerative Disease and an Amyotrophic Lateral Sclerosis-Like Syndrome , 2007, Drug safety.

[17]  Johan Hopstadius,et al.  Safety surveillance of longitudinal databases: methodological considerations , 2011, Pharmacoepidemiology and drug safety.

[18]  M. Schuemie,et al.  Combining electronic healthcare databases in Europe to allow for large‐scale drug safety monitoring: the EU‐ADR Project , 2011, Pharmacoepidemiology and drug safety.

[20]  Joseph M. Tonning,et al.  Pharmacovigilance in the 21st Century: New Systematic Tools for an Old Problem , 2004, Pharmacotherapy.

[21]  Z. Bankowski,et al.  Council for International Organizations of Medical Sciences , 1991 .

[22]  Manfred Hauben,et al.  Defining ‘Signal’ and its Subtypes in Pharmacovigilance Based on a Systematic Review of Previous Definitions , 2009, Drug safety.

[23]  S. Nissen,et al.  An Updated Meta-analysis of Risk for Myocardial Infarction and Cardiovascular Mortality , 2010 .

[24]  A. Pariente,et al.  Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? , 2009, Pharmacoepidemiology and drug safety.

[25]  R. O’Neill,et al.  Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA’s Spontaneous Reports Database , 2002, Drug safety.