Evaluation of signal detection methods for use in prospective post licensure medical product safety surveillance

The Sentinel Initiative aims to develop a national, electronic network to link data on 100 million patients from multiple existing health care data systems by 2012 in order to conduct post-licensur e safety monitoring of FDA-regulated medical products. A critical component of this Initiative is determining what statistical methods can bes t be employed within this framework to accurately, robustly, and flexibly detect safety problems. In this report, we review existing signal detection methods for possible use in the Sentinel Initiative. We statistically and clinicall y describe each method, summarize prior and current uses 0 f each method, and assess what is kn own about each method's robustness across different data sources and each method's flexibility to accommodate different signaling threshold levels and different types of adverse event outcomes. Recommendations for pursuing the most promising signal detection methods for use in the Sentinel Initiative are given.

[1]  V T Farewell,et al.  Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts , 2003, Statistical methods in medical research.

[2]  R J Cook,et al.  Monitoring paired binary surgical outcomes using cumulative sum charts. , 1999, Statistics in medicine.

[3]  Daniel L Gillen,et al.  Bayesian evaluation of group sequential clinical trial designs , 2007, Statistics in medicine.

[4]  R. Kronmal,et al.  Multi-Ethnic Study of Atherosclerosis: objectives and design. , 2002, American journal of epidemiology.

[5]  A. Tsiatis,et al.  Approximately optimal one-parameter boundaries for group sequential trials. , 1987, Biometrics.

[6]  A F BARTHOLOMAY.  The sequential probability ratio test applied to the design of clinical experiments. , 1957, The New England journal of medicine.

[7]  G. Apostolakis,et al.  Exploration of a Bayesian Updating Methodology to Monitor the Safety of Interventional Cardiovascular Procedures , 2004, Medical decision making : an international journal of the Society for Medical Decision Making.

[8]  William DuMouchel,et al.  Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System , 1999 .

[9]  S. Evans,et al.  Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports , 2001, Pharmacoepidemiology and drug safety.

[10]  Tom Treasure,et al.  Risk-adjusted sequential probability ratio tests: applications to Bristol, Shipman and adult cardiac surgery. , 2003, International journal for quality in health care : journal of the International Society for Quality in Health Care.

[11]  Zachary G. Stoumbos,et al.  Monitoring a Proportion Using CUSUM and SPRT Control Charts , 2001 .

[12]  Andrew Bate,et al.  Bayesian Confidence Propagation Neural Network , 2007, Drug safety.

[13]  M. Kulldorff,et al.  Early detection of adverse drug events within population‐based health networks: application of sequential testing methods , 2007, Pharmacoepidemiology and drug safety.

[14]  Nilanjan Chatterjee,et al.  Design and analysis of two‐phase studies with binary outcome applied to Wilms tumour prognosis , 1999 .

[15]  R. Kronmal,et al.  The Cardiovascular Health Study: design and rationale. , 1991, Annals of epidemiology.

[16]  B. Turnbull,et al.  Group Sequential Methods with Applications to Clinical Trials , 1999 .

[17]  R J Cook,et al.  Risk-Adjusted Monitoring of Binary Surgical Outcomes , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[18]  P. O'Brien,et al.  A multiple testing procedure for clinical trials. , 1979, Biometrics.

[19]  P. Armitage,et al.  Sequential tests in prophylactic and therapeutic trials. , 1954, The Quarterly journal of medicine.

[20]  Jeffrey R Curtis,et al.  Adaptation of Bayesian Data Mining Algorithms to Longitudinal Claims Data: Coxib Safety as an Example , 2008, Medical care.

[21]  K. Kapur,et al.  Mixed‐effects Poisson regression analysis of adverse event reports: The relationship between antidepressants and suicide , 2008, Statistics in medicine.

[22]  William DuMouchel,et al.  Empirical bayes screening for multi-item associations , 2001, KDD '01.

[23]  S. Pocock.  Group sequential methods in the design and analysis of clinical trials , 1977 .

[24]  M. Kulldorff,et al.  A Maximized Sequential Probability Ratio Test for Drug and Vaccine Safety Surveillance , 2011 .

[25]  W. Irish,et al.  Detecting signals of opioid analgesic abuse: application of a spatial mixed effect poisson regression model using data from a network of poison control centers , 2008, Pharmacoepidemiology and drug safety.

[26]  G Gallus,et al.  On surveillance methods for congenital malformations. , 1986, Statistics in medicine.

[27]  Isao Yoshimura,et al.  Criteria Revision and Performance Comparison of Three Methods of Signal Detection Applied to the Spontaneous Reporting Database of a Pharmaceutical Manufacturer , 2007, Drug safety.

[28]  Manfred Hauben,et al.  Safety Related Drug-Labelling Changes , 2004, Drug safety.

[29]  V. Rich.  Personal communication , 1989, Nature.

[30]  S. Steiner,et al.  Monitoring surgical performance using risk-adjusted cumulative sum charts. , 2000, Biostatistics.

[31]  M. Lindquist,et al.  A Data Mining Approach for Signal Detection and Analysis , 2002, Drug safety.

[32]  Carl-Erik Särndal,et al.  Generalized Raking Procedures in Survey Sampling , 1993 .

[33]  Lucila Ohno-Machado,et al.  Research Paper: Monitoring Device Safety in Interventional Cardiology , 2006, J. Am. Medical Informatics Assoc..

[34]  R. O’Neill,et al.  Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA’s Spontaneous Reports Database , 2002, Drug safety.

[35]  A Bate,et al.  The application of knowledge discovery in databases to post‐marketing drug safety: example of the WHO database , 2008, Fundamental & clinical pharmacology.

[36]  David Bock,et al.  A review and discussion of prospective statistical surveillance in public health , 2003 .

[37]  M. Braun,et al.  Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. , 2001, Vaccine.

[38]  G Gallus,et al.  Monitoring surgical and medical outcomes: the Bernoulli cumulative SUM chart. A novel application to assess clinical interventions , 2005, Postgraduate Medical Journal.

[39]  M R de Leval,et al.  Analysis of a cluster of surgical failures. Application to a series of neonatal arterial switch operations. , 1994, The Journal of thoracic and cardiovascular surgery.

[40]  A. Bate,et al.  A Bayesian neural network method for adverse drug reaction signal generation , 1998, European Journal of Clinical Pharmacology.

[41]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[42]  R. Khan,et al.  Sequential Tests of Statistical Hypotheses. , 1972 .

[43]  G. Moustakides.  Optimal stopping times for detecting changes in distributions , 1986 .

[44]  Robert Ball,et al.  Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance data. , 2005, Statistics in medicine.

[45]  Richard Platt,et al.  Early adverse drug event signal detection within population‐based health networks using sequential methods: key methodologic considerations , 2009, Pharmacoepidemiology and drug safety.

[46]  Stephanie Chung,et al.  Postmarketing surveillance of potentially fatal reactions to oncology drugs: potential utility of two signal-detection algorithms , 2004, European Journal of Clinical Pharmacology.

[47]  G Apostolakis,et al.  Modeling the detection rates of fires in nuclear plants: development and application of a methodology for treating imprecise evidence. , 1986, Risk analysis : an official publication of the Society for Risk Analysis.

[48]  P. Armitage,et al.  Sequential methods in clinical trials. , 1958, American journal of public health and the nation's health.

[49]  Robert L Davis,et al.  Real-Time Vaccine Safety Surveillance for the Early Detection of Adverse Events , 2007, Medical care.

[50]  D. Burwen,et al.  Comparing data mining methods on the VAERS database , 2005, Pharmacoepidemiology and drug safety.

[51]  Lucila Ohno-Machado,et al.  Validation of an Automated Safety Surveillance System with Prospective, Randomized Trial Data , 2009, Medical decision making : an international journal of the Society for Medical Decision Making.

[52]  Robert Ball,et al.  Effects of Stratification on Data Mining in the US Vaccine Adverse Event Reporting System (VAERS) , 2008, Drug safety.

[53]  Robert T. Chen,et al.  Active Surveillance of Vaccine Safety: A System to Detect Early Signs of Adverse Events , 2005, Epidemiology.