Performance of Pharmacovigilance Signal‐Detection Algorithms for the FDA Adverse Event Reporting System

Signal‐detection algorithms (SDAs) are recognized as vital tools in pharmacovigilance. However, their performance characteristics are generally unknown. By leveraging a unique gold standard recently made public by the Observational Medical Outcomes Partnership (OMOP) and by conducting a unique systematic evaluation, we provide new insights into the diagnostic potential and characteristics of SDAs that are routinely applied to the US Food and Drug Administration (FDA) Adverse Event Reporting System (AERS). We find that SDAs can attain reasonable predictive accuracy in signaling adverse events. Two performance classes emerge, indicating that the class of approaches that address confounding and masking effects benefits safety surveillance. Our study shows that not all events are equally detectable, suggesting that specific events might be monitored more effectively using other data sources. We provide performance guidelines for several operating scenarios to inform the trade‐off between sensitivity and specificity for specific use cases. We also propose an approach and demonstrate its application in identifying optimal signaling thresholds, given specific misclassification tolerances.

[1]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[2]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[3]  J. Sengupta.  The Nonparametric Approach , 1989 .

[4]  L. Meskin.  Proceed with caution. , 1998, Journal of the American Dental Association.

[5]  William DuMouchel,et al.  Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System , 1999 .

[6]  S. Evans,et al.  Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports , 2001, Pharmacoepidemiology and drug safety.

[7]  M. Lindquist,et al.  A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions , 2002, Pharmacoepidemiology and drug safety.

[8]  R. O’Neill,et al.  Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA’s Spontaneous Reports Database , 2002, Drug safety.

[9]  E. Brown,et al.  Methods and Pitfalls in Searching Drug Safety Databases Utilising the Medical Dictionary for Regulatory Activities (MedDRA)1 , 2003, Drug safety.

[10]  W. Rayford,et al.  Early detection of prostate cancer in African-American men through use of multiple biomarkers: human kallikrein 2 (hK2), prostate-specific antigen (PSA), and free PSA (fPSA) , 2004, Prostate Cancer and Prostatic Diseases.

[11]  N. Hartnell,et al.  Replication of the Weber Effect Using Postmarketing Adverse Event Reports Voluntarily Submitted to the United States Food and Drug Administration , 2004, Pharmacotherapy.

[12]  Daisuke Koide,et al.  Comparison of data mining methodologies using Japanese spontaneous reports , 2004, Pharmacoepidemiology and drug safety.

[13]  Joseph M. Tonning,et al.  Pharmacovigilance in the 21st Century: New Systematic Tools for an Old Problem , 2004, Pharmacotherapy.

[14]  Antoine Egberts,et al.  The reporting odds ratio versus the proportional reporting ratio: ‘deuce’ , 2004, Pharmacoepidemiology and Drug Safety.

[15]  A. Venot,et al.  Appraisal of the MedDRA Conceptual Structure for Describing and Grouping Adverse Drug Reactions , 2005, Drug safety.

[16]  J. Tisdale,et al.  Drug-Induced Diseases: Prevention, Detection, and Management , 2005 .

[17]  D. Madigan,et al.  The role of data mining in pharmacovigilance , 2005, Expert opinion on drug safety.

[18]  C. D'Orsi,et al.  Diagnostic Performance of Digital Versus Film Mammography for Breast-Cancer Screening , 2005, The New England journal of medicine.

[19]  D. Burwen,et al.  Comparing data mining methods on the VAERS database , 2005, Pharmacoepidemiology and drug safety.

[20]  Joseph M. Tonning,et al.  Perspectives on the Use of Data Mining in Pharmacovigilance , 2005, Drug safety.

[21]  Manfred Hauben,et al.  Potential Utility of Data‐Mining Algorithms for Early Detection of Potentially Fatal/Disabling Adverse Drug Reactions: A Retrospective Evaluation , 2005, Journal of clinical pharmacology.

[22]  Joseph M. Tonning,et al.  British Journal of Clinical Pharmacology Letter to the Editors Reply: the Evaluation of Data Mining Methods for the Simultaneous and Systematic Detection of Safety Signals in Large Databases: Lessons to Be Learned , 2022 .

[23]  W. DuMouchel,et al.  Comparative Performance of Two Quantitative Safety Signalling Methods , 2006, Drug safety.

[24]  John A. Clark,et al.  Statin safety: an appraisal from the adverse event reporting system. , 2006, The American journal of cardiology.

[25]  N. Perkins,et al.  The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. , 2006, American journal of epidemiology.

[26]  M. Hauben,et al.  Data mining for signals in spontaneous reporting databases: proceed with caution , 2007, Pharmacoepidemiology and drug safety.

[27]  A L Gould,et al.  An Evaluation of Computer‐Aided Disproportionality Analysis for Post‐Marketing Signal Detection , 2007, Clinical pharmacology and therapeutics.

[28]  Stephanie J. Reisinger,et al.  Using Data Mining to Predict Safety Actions from FDA Adverse Event Reporting System Data , 2007 .

[29]  W. DuMouchel,et al.  Novel Statistical Tools for Monitoring the Safety of Marketed Drugs , 2007, Clinical pharmacology and therapeutics.

[30]  William DuMouchel,et al.  Antipsychotics, glycemic disorders, and life-threatening diabetic events: a Bayesian data-mining analysis of the FDA adverse event reporting system (1968-2004). , 2008, Annals of clinical psychiatry : official journal of the American Academy of Clinical Psychiatrists.

[31]  Johan Hopstadius,et al.  Impact of Stratification on Adverse Drug Reaction Surveillance , 2008, Drug safety.

[32]  David Madigan,et al.  An Evaluation of Three Signal-Detection Algorithms Using a Highly Inclusive Reference Event Database , 2009, Drug safety.

[33]  R. Platt,et al.  The new Sentinel Network--improving the evidence of medical-product safety. , 2009, The New England journal of medicine.

[34]  A. Pariente,et al.  Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? , 2009, Pharmacoepidemiology and drug safety.

[35]  A. Bate,et al.  Quantitative signal detection using spontaneous ADR reporting , 2009, Pharmacoepidemiology and drug safety.

[36]  A Bate,et al.  Decision support methods for the detection of adverse events in post-marketing data. , 2009, Drug discovery today.

[37]  Sebastian Schneeweiss,et al.  A basic study design for expedited safety signal evaluation based on electronic healthcare data , 2010, Pharmacoepidemiology and drug safety.

[38]  Jim Slattery,et al.  Validation of Statistical Signal Detection Procedures in EudraVigilance Post-Authorization Data , 2010, Drug safety.

[39]  J. Overhage,et al.  Advancing the Science for Active Surveillance: Rationale and Design for the Observational Medical Outcomes Partnership , 2010, Annals of Internal Medicine.

[40]  R. Tiwari,et al.  A Likelihood Ratio Test Based Method for Signal Detection With Application to FDA’s Drug Safety Data , 2011 .

[41]  P Ryan,et al.  Novel Data‐Mining Methodologies for Adverse Drug Event Discovery and Analysis , 2012, Clinical pharmacology and therapeutics.

[42]  D. Madigan,et al.  Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership , 2012, Statistics in medicine.

[43]  Conny Berlin,et al.  Are all quantitative postmarketing signal detection methods equal? Performance characteristics of logistic regression and Multi‐item Gamma Poisson Shrinker , 2012, Pharmacoepidemiology and drug safety.