An Evaluation of Three Signal-Detection Algorithms Using a Highly Inclusive Reference Event Database

AbstractBackground: Pharmacovigilance data-mining algorithms (DMAs) are known to generate significant numbers of false-positive signals of disproportionate reporting (SDRs), using various standards to define the terms ‘true positive’ and ‘false positive’. Objective: To construct a highly inclusive reference event database of reported adverse events for a limited set of drugs, and to utilize that database to evaluate three DMAs for their overall yield of scientifically supported adverse drug effects, with an emphasis on ascertaining false-positive rates as defined by matching to the database, and to assess the overlap among SDRs detected by various DMAs. Methods: A sample of 35 drugs approved by the US FDA between 2000 and 2004 was selected, including three drugs added to cover therapeutic categories not included in the original sample. We compiled a reference event database of adverse event information for these drugs from historical and current US prescribing information, from peer-reviewed literature covering 1999 through March 2006, from regulatory actions announced by the FDA and from adverse event listings in the British National Formulary. Every adverse event mentioned in these sources was entered into the database, even those with minimal evidence for causality. To provide some selectivity regarding causality, each entry was assigned a level of evidence based on the source of the information, using rules developed by the authors. Using the FDA adverse event reporting system data for 2002 through 2005, SDRs were identified for each drug using three DMAs: an urn-model based algorithm, the Gamma Poisson Shrinker (GPS) and proportional reporting ratio (PRR), using previously published signalling thresholds. The absolute number and fraction of SDRs matching the reference event database at each level of evidence was determined for each report source and the data-mining method. Overlap of the SDR lists among the various methods and report sources was tabulated as well. Results: The GPS algorithm had the lowest overall yield of SDRs (763), with the highest fraction of events matching the reference event database (89 SDRs, 11.7%), excluding events described in the prescribing information at the time of drug approval. The urn model yielded more SDRs (1562), with a non-significantly lower fraction matching (175 SDRs, 11.2%). PRR detected still more SDRs (3616), but with a lower fraction matching (296 SDRs, 8.2%). In terms of overlap of SDRs among algorithms, PRR uniquely detected the highest number of SDRs (2231, with 144, or 6.5%, matching), followed by the urn model (212, with 26, or 12.3%, matching) and then GPS (0 SDRs uniquely detected). Conclusions: The three DMAs studied offer significantly different tradeoffs between the number of SDRs detected and the degree to which those SDRs are supported by external evidence. Those differences may reflect choices of detection thresholds as well as features of the algorithms themselves. For all three algorithms, there is a substantial fraction of SDRs for which no external supporting evidence can be found, even when a highly inclusive search for such evidence is conducted.

[1]  Joseph M. Tonning,et al.  Perspectives on the Use of Data Mining in Pharmacovigilance , 2005, Drug safety.

[2]  A. B. Prasad,et al.  British National Formulary , 1994 .

[3]  J. Venulet,et al.  Standardized assessment of drug-adverse reaction associations--rationale and experience. , 1980, International journal of clinical pharmacology, therapy, and toxicology.

[4]  Manfred Hauben,et al.  Trimethoprim-induced hyperkalaemia -- lessons in data mining. , 2004, British journal of clinical pharmacology.

[5]  Stephanie J. Reisinger,et al.  Using Data Mining to Predict Safety Actions from FDA Adverse Event Reporting System Data , 2007 .

[6]  William DuMouchel,et al.  Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System , 1999 .

[7]  S. Evans,et al.  Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports , 2001, Pharmacoepidemiology and drug safety.

[8]  M. Lindquist,et al.  A Retrospective Evaluation of a Data Mining Approach to Aid Finding New Adverse Drug Reaction Signals in the WHO International Database , 2000, Drug safety.

[9]  J. Reynolds.  Martindale : the extra pharmacopoeia , 1972 .

[10]  Darcy Wolfman,et al.  Satisfaction of search in osteoradiology. , 2000, AJR. American journal of roentgenology.

[11]  A Lawrence Gould,et al.  Practical pharmacovigilance analysis strategies. , 2003, Pharmacoepidemiology and drug safety.

[12]  M. Lindquist,et al.  Signal Selection and Follow-Up in Pharmacovigilance , 2002, Drug safety.

[13]  M. Hauben,et al.  Time‐to‐Signal Comparison for Drug Safety Data‐Mining Algorithms vs. Traditional Signaling Criteria , 2009, Clinical pharmacology and therapeutics.

[14]  E. Brown,et al.  The Medical Dictionary for Regulatory Activities (MedDRA) , 1999, Drug safety.

[15]  D. Greenblatt,et al.  A method for estimating the probability of adverse drug reactions , 1981, Clinical pharmacology and therapeutics.

[16]  D. Madigan,et al.  The role of data mining in pharmacovigilance , 2005, Expert opinion on drug safety.

[17]  Robert Ball,et al.  Effects of Stratification on Data Mining in the US Vaccine Adverse Event Reporting System (VAERS) , 2008, Drug safety.

[18]  J. Aronson,et al.  Gold Standards in Pharmacovigilance , 2006 .

[19]  L Lasagna,et al.  Toward the operational identification of adverse drug reactions , 1977, Clinical pharmacology and therapeutics.

[20]  Manfred Hauben,et al.  Safety Related Drug-Labelling Changes , 2004, Drug safety.

[21]  Communication of findings in pharmacovigilance: use of the term “signal” and the need for precision in its use , 2005, European Journal of Clinical Pharmacology.

[22]  Jeffrey K Aronson,et al.  Anecdotes as evidence , 2003, BMJ : British Medical Journal.

[23]  Manfred Hauben,et al.  Signal detection in pharmacovigilance: empirical evaluation of data mining tools , 2005, Pharmacoepidemiology and drug safety.

[24]  Manfred Hauben,et al.  What Counts in Data Mining? , 2006, Drug safety.