Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance data.

The objective of the work described in this paper is to develop a means for characterizing the validity of an empirical methodology for detecting signals potentially related to complicated adverse event (AE) coding terms in multidimensional public health surveillance data. The signal detection tool under evaluation is the multi-item gamma Poisson shrinkage (MGPS) estimation program. We were interested in its potential application to passive surveillance system monitoring, to screen for 'signals' of complicated adverse event coding terms (AE terms) in complex and noisy data. The research was to design and produce a flexible and user-friendly utility for probabilistically defining complicated signals in a database, iterating large numbers of applications of the MGPS detection algorithm and establishing proportions of correct detection events. We sought to establish the specificity of the MGPS by developing a random background using a gradient that ranged from rigorous (but not very relevant) to relevant (but noisy). To establish the sensitivity, signals were defined based on recognized public health issues of interest (such as the introduction of a new vaccine into the population). Methods of representing a signal included a simple pair-wise association consisting of a new vaccine and one AE term, as well as a more realistic complex of multiple AE terms comprising a 'syndrome'. A web application has been developed to create and insert signals with user-defined probabilities in multiple iterations of simulated random background data. Three forms of simulated data based on the vaccine adverse event reporting system (VAERS) cumulative spontaneous database were defined to serve as background noise against which to contrast introduced vaccine adverse event signals: (1) completely random associations between vaccines and AE terms, (2) random associations of vaccine sets and AE term sets preserving naturally observed vaccine co-occurrences and AE term co-occurrences and (3) samples from the actual VAERS data as reported. Rates of detection by the MGPS algorithm can be established for specific signal patterns at varying probabilistic intensities in a choice of random background data forms. Knowing these rates is important for determining the degree of response to an MGPS signal detection event in 'live' data.

[1]  M. Lindquist,et al.  Signal Selection and Follow-Up in Pharmacovigilance , 2002, Drug safety.

[2]  Eileen Salinsky.  Will the nation be ready for the next bioterrorism attack? Mending gaps in the public health infrastructure. , 2002, NHPF issue brief.

[3]  David Bock,et al.  A review and discussion of prospective statistical surveillance in public health , 2003 .

[4]  R. Meyboom,et al.  Use of Measures of Disproportionality in Pharmacovigilance , 2002, Drug safety.

[5]  S. Evans,et al.  Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports , 2001, Pharmacoepidemiology and drug safety.

[6]  William DuMouchel,et al.  Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting System , 1999 .

[7]  G. D. Williamson,et al.  A monitoring system for detecting aberrations in public health surveillance reports. , 1999, Statistics in medicine.

[8]  Lisa J. Trigg,et al.  Roundtable on Bioterrorism Detection , 2002 .

[9]  M. Braun,et al.  Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. , 2001, Vaccine.

[10]  R. O’Neill,et al.  Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA’s Spontaneous Reports Database , 2002, Drug safety.

[11]  William DuMouchel,et al.  Empirical Bayesian data mining for discovering patterns in post-marketing drug safety , 2003, KDD '03.

[12]  Dean F. Sittig,et al.  The emerging science of very early detection of disease outbreaks. , 2001, Journal of public health management and practice : JPHMP.

[13]  Robert T. O'Neill,et al.  Some US Food and Drug Administration Perspectives on Data Mining for Pediatric Safety Assessment , 2001 .

[14]  Kathleen Stratton,et al.  Research Strategies for Assessing Adverse Events Associated with Vaccines , 1994 .

[15]  William DuMouchel,et al.  Empirical bayes screening for multi-item associations , 2001, KDD '01.

[16]  Nick Andrews,et al.  A Statistical Algorithm for the Early Detection of Outbreaks of Infectious Disease , 1996 .