Utility of Composite Reference Standards and Latent Class Analysis in Evaluating the Clinical Accuracy of Diagnostic Tests for Pertussis

ABSTRACT Numerous evaluations of the clinical sensitivity and specificity of PCR and serologic assays for Bordetella pertussis have been hampered by the low sensitivity of culture, the gold standard test, which leads to biased accuracy estimates. The bias can be reduced by using statistical approaches such as the composite reference standard (CRS) (e.g., positive if culture or serology positive; negative otherwise) or latent class analysis (LCA), an internal reference standard based on a statistical model. We illustrated the benefits of the CRS and LCA approaches by reanalyzing data from a 1995 to 1996 study of cough illness among 212 patients. The accuracy of PCR in this study was evaluated using three reference standards: culture, CRS, and LCA. Using specimens obtained 0 to 34 days after cough onset, estimates of the sensitivity of PCR obtained using CRS (47%) and LCA (34%) were lower than the culture-based estimate (62%). The CRS and LCA approaches, which utilized more than one diagnostic marker of pertussis, likely produced more accurate reference standards than culture alone. In general, the CRS approach is simple, with a well-defined disease status. LCA requires statistical modeling but incorporates more indicators of disease than CRS. When three or more indicators of pertussis are available, these approaches should be used in evaluations of pertussis diagnostic tests.

[1]  C. R. Westwick Item analysis. , 1976, The Journal of nursing education.

[2]  A. Hadgu,et al.  Evaluation of Nucleic Acid Amplification Tests in the Absence of a Perfect Gold-Standard Test: A Review of the Statistical and Epidemiologic Issues , 2005, Epidemiology.

[3]  J. Mertsola,et al.  Defining Pertussis Epidemiology: Clinical, Microbiologic and Serologic Perspectives , 2005, The Pediatric infectious disease journal.

[4]  J. Schwartz,et al.  Evaluating diagnostic tests , 1986, Journal of General Internal Medicine.

[5]  M S Pepe,et al.  Using a combination of reference tests to assess the accuracy of a new diagnostic test. , 1999, Statistics in medicine.

[6]  Xiao-Hua Zhou,et al.  Statistical Methods in Diagnostic Medicine , 2002 .

[7]  C. Black,et al.  Evaluation of Bias in Diagnostic-Test Sensitivity and Specificity Estimates Computed by Discrepant Analysis , 1998, Journal of Clinical Microbiology.

[8]  A. Van Rie,et al.  Diagnosis of pertussis: a historical review and recent developments , 2006, Expert review of molecular diagnostics.

[9]  J. Hagenaars,et al.  Applied Latent Class Analysis , 2003 .

[10]  George F. Reed,et al.  Use of Coefficient of Variation in Assessing Variability of Quantitative Assays , 2002, Clinical and Vaccine Immunology.

[11]  Bernard,et al.  Evaluation and validation of a monoclonal immunofluorescent reagent for direct detection of Bordetella pertussis , 1995, Journal of clinical microbiology.

[12]  Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests ; Draft Guidance for Industry and FDA Reviewers Draft Guidance-Not for Implementation , 2003 .

[13]  W. Grove Statistical Methods for Rates and Proportions, 2nd ed , 1981 .

[14]  G. Ehrlich,et al.  Multiplex PCR-based assay for detection of Bordetella pertussis in nasopharyngeal swab specimens , 1996, Journal of clinical microbiology.

[15]  M Staquet,et al.  Methodology for the assessment of new dichotomous diagnostic tests. , 1981, Journal of chronic diseases.

[16]  A. Fluit,et al.  False-Positive Results and Contamination in Nucleic Acid Amplification Assays: Suggestions for a Prevent and Destroy Strategy , 2004, European Journal of Clinical Microbiology and Infectious Diseases.

[17]  S D Walter,et al.  Effects of dependent errors in the assessment of diagnostic test performance. , 1997, Statistics in medicine.

[18]  A. Feinstein,et al.  Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. , 1978, The New England journal of medicine.

[19]  Els Goetghebeur,et al.  Diagnostic test analyses in search of their gold standard: latent class analyses with random effects , 2000, Statistical methods in medical research.

[20]  J. Sellors,et al.  Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis. , 1998, Statistics in medicine.

[21]  L. Joseph,et al.  Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. , 1995, American journal of epidemiology.

[22]  J. A. Hunt,et al.  Population-based incidence of pertussis among adolescents and adults, Minnesota, 1995-1996. , 2001, The Journal of infectious diseases.

[23]  V. Caro,et al.  Nucleic Acid Amplification Tests for Diagnosis of Bordetella Infections , 2005, Journal of Clinical Microbiology.

[24]  F. Müller,et al.  Laboratory diagnosis of pertussis: state of the art in 1997 , 1997, Journal of clinical microbiology.

[25]  J. Fleiss Statistical methods for rates and proportions , 1974 .

[26]  W. Miller Can we do better than discrepant analysis for new diagnostic test evaluation? , 1998, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[27]  J. Schachter Editorial Response: Two Different Worlds We Live In , 1998 .

[28]  F. Mooi,et al.  Polymerase chain reaction assay for pertussis: simultaneous detection and discrimination of Bordetella pertussis and Bordetella parapertussis , 1993, Journal of clinical microbiology.

[29]  J. Vermunt,et al.  Latent Gold 4.0 User's Guide , 2005 .

[30]  C. Welinder-Olsson,et al.  Evaluation of PCR for Diagnosis of Bordetella pertussis and Bordetella parapertussisInfections , 1998, Journal of Clinical Microbiology.

[31]  A. Hadgu The discrepancy in discrepant analysis , 1996, The Lancet.

[32]  S D Walter,et al.  Estimation of test sensitivity and specificity when disease confirmation is limited to positive results. , 1999, Epidemiology.

[33]  A. Schuchat,et al.  Classical and latent class analysis evaluation of sputum polymerase chain reaction and urine antigen testing for diagnosis of pneumococcal pneumonia in adults. , 2003, The Journal of infectious diseases.

[34]  J. Taranger,et al.  Environmental contamination leading to false-positive polymerase chain reaction for pertussis. , 1994, The Pediatric infectious disease journal.

[35]  A. Golaz,et al.  Issues Associated with and Recommendations for Using PCR To Detect Outbreaks of Pertussis , 2002, Journal of Clinical Microbiology.

[36]  S D Walter,et al.  Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. , 1988, Journal of clinical epidemiology.

[37]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[38]  R E Johnson,et al.  In defense of discrepant analysis. , 2001, Journal of clinical epidemiology.

[39]  M. Loeffelholz,et al.  Comparison of PCR, Culture, and Direct Fluorescent-Antibody Testing for Detection of Bordetella pertussis , 1999, Journal of Clinical Microbiology.

[40]  R L Guyer,et al.  The molecule of the year. , 1989 .

[41]  K. Edwards,et al.  Establishment of Diagnostic Cutoff Points for Levels of Serum Antibodies to Pertussis Toxin, Filamentous Hemagglutinin, and Fimbriae in Adolescents and Adults in the United States , 2004, Clinical Diagnostic Laboratory Immunology.

[42]  Outbreaks of respiratory illness mistakenly attributed to pertussis--New Hampshire, Massachusetts, and Tennessee, 2004-2006. , 2007, MMWR. Morbidity and mortality weekly report.

[43]  W. Blattner,et al.  A method for predicting individual HIV infection status in the absence of clinical information. , 1988, AIDS research and human retroviruses.

[44]  S. Hui,et al.  Evaluation of diagnostic tests without gold standards , 1998, Statistical methods in medical research.

[45]  A. McAdam,et al.  Discrepant Analysis: How Can We Test a Test? , 2000, Journal of Clinical Microbiology.

[46]  H. Lipman,et al.  Quantifying the bias associated with use of discrepant analysis. , 1998, Clinical chemistry.

[47]  T L Chorba,et al.  Case definitions for public health surveillance. , 1990, MMWR. Recommendations and reports : Morbidity and mortality weekly report. Recommendations and reports.

[48]  Scott Zeger,et al.  Methods for evaluating the performance of diagnostic tests in the absence of a gold standard: a latent class model approach , 2002, Statistics in medicine.

[49]  D E Koshland,et al.  The Molecule of the Year. , 1989, Science.

[50]  E Goetghebeur,et al.  The potential of Latent Class Analysis in diagnostic test validation for canine Leishmania infantum infection , 1999, Epidemiology and Infection.

[51]  I. Onorato,et al.  Laboratory diagnosis of pertussis: the state of the art , 1987, The Pediatric infectious disease journal.

[52]  L. Joseph,et al.  Bayesian Approaches to Modeling the Conditional Dependence Between Multiple Diagnostic Tests , 2001, Biometrics.

[53]  J J Gart,et al.  Comparison of a screening test and a reference test in epidemiologic studies. I. Indices of agreement and their relation to prevalence. , 1966, American journal of epidemiology.

[54]  P. Qiu The Statistical Evaluation of Medical Tests for Classification and Prediction , 2005 .

[55]  D M Hawkins,et al.  Some issues in resolution of diagnostic tests using an imperfect gold standard , 2001, Statistics in medicine.

[56]  L. A. Thibodeaul Evaluating Diagnostic Tests , 1981 .

[57]  J. Gart,et al.  Comparison of a screening test and a reference test in epidemiologic studies. II. A probabilistic model for the comparison of diagnostic tests. , 1966, American journal of epidemiology.

[58]  W. Miller,et al.  Bias in discrepant analysis: when two wrongs don't make a right. , 1998, Journal of clinical epidemiology.

[59]  A. Bollen,et al.  Recommendations for use of the polymerase chain reaction in the diagnosis of Bordetella pertussis infections. , 1994, Journal of medical microbiology.

[60]  Bias in the evaluation of DNA-amplification tests for detecting Chlamydia trachomatis. , 1997, Statistics in medicine.

[61]  P. Strebel,et al.  Pertussis in Missouri: evaluation of nasopharyngeal culture, direct fluorescent antibody testing, and clinical case definitions in the diagnosis of pertussis. , 1993, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[62]  P N Valenstein,et al.  Evaluating diagnostic tests with imperfect standards. , 1990, American journal of clinical pathology.

[63]  J. Vermunt Latent Class Models , 2004 .

[64]  Allan L. McCutcheon,et al.  Applied Latent Class Analysis: Basic Concepts and Procedures in Single- and Multiple-Group Latent Class Analysis , 2002 .

[65]  S. Halperin,et al.  Pertussis of adults and infants. , 2002, The Lancet. Infectious diseases.