Accuracy of the Papanicolaou Test in Screening for and Follow-up of Cervical Cytologic Abnormalities

Since the implementation of widespread screening with the Papanicolaou (Pap) test, rates of cervical cancer in the United States have decreased from 14.2 per 100 000 in 1973 to 7.8 per 100 000 in 1994. Nevertheless, cervical cancer is still the ninth-leading cause of cancer deaths among U.S. women (1). Most of these deaths occur in women who have never had a Pap test, but some occur in women who recently received negative test results. Approximately two thirds of false-negative results are caused by sampling error, and the rest are caused by detection error. Sampling error occurs when abnormal cells are not collected or are not transferred to the Pap slide, and detection error occurs when abnormal cells on the Pap slide are missed or misinterpreted. The most common sampling error is lack of cells from the cervical transformation zone. To reduce sampling error, an endocervical cytobrush and a spatula can be used instead of a cotton swab. However, a recent meta-analysis found that the Pap test did not differ in sensitivity or specificity when different sampling devices were used (2). The Food and Drug Administration (FDA) has approved another potential solution: liquid-based monolayer preparation (ThinPrep, Cytyc Corp., Boxborough, Massachusetts). With this technique, the sample is collected as in the conventional Pap test, but cells are then placed in a fixative solution. The cells are dispersed, collected onto a filter, and transferred to a microscopic slide for interpretation. Because samples are fixed immediately after collection, fewer cellular morphologic artifacts occur. Fewer cells on the slide are obscured because the process reduces the amounts of other sampled material, such as blood and mucus, and deposits cells on the slide in a monolayer. To reduce detection error, some researchers advocate rescreening slides initially reported to be normal. The Clinical Laboratory Improvement Amendments of 1988 mandate rescreening of a 10% random sample of normal slides as a quality assurance measure. Rescreening can also be performed on a higher proportion of slides by using computerized technologies. The FDA has approved two such systems, one that is algorithm-based (AutoPap QC System, TriPath Imaging, Inc., Redmond, Washington), and one that uses neural networks (PAPNET, Neuromedical Systems, Inc., Suffern, New York). PAPNET uses neural network computerized imaging of Papanicolaou smear slides to identify cells or clusters of cells that require review; it then displays up to 128 images per slide that are likely to contain abnormalities. A cytotechnologist reviews these images and decides whether to review the actual slide using light microscopy. AutoPap uses its algorithm-based decision-making technology to identify slides that exceed a certain threshold for the likelihood of abnormal cells. The laboratory can select different thresholds, corresponding to 10%, 15%, or 20% review rates. In contrast to random rescreening, AutoPap selects a sample of slides that is enriched with abnormalities, thereby including most of the slides that contain abnormalities missed by manual screening. Another approach to reducing detection error is improving the sensitivity of the initial screening step. The FDA has recently approved a new method (AutoPap Primary Screening System, TriPath Imaging) for this indication. AutoPap Primary Screening System uses proprietary computerized algorithms to identify slides that exceed a certain threshold for the likelihood of abnormal cells. A cytotechnologist then reviews these slides. The system allows laboratories to concentrate on the 75% of slides that most likely contain abnormal cells while immediately archiving the remainder. Sampling and detection errors are reduced when Pap test screening is repeated frequently. However, cost-effectiveness analyses have concluded that if persons are screened more than every 3 years, cost-effectiveness ratios exceed $50 000 per life-year saved (3, 4). Precise estimates of cytologic test sensitivity and specificity are important because they may be used to determine policy decisions, such as recommendations for optimal frequency of screening, management of mild abnormalities, and use of newer methods. Our primary objective was to systematically review the operating characteristics of conventional and new methods (computer screening and monolayer slide preparation) of Pap testing in the detection of cervical cancer and its precursors. We also evaluated test performance among women with previous cytologic abnormalities. The Agency for Healthcare Research and Quality (AHRQ), under contract to Duke University (Durham, North Carolina), funded the study. An AHRQ-approved advisory panel assisted in the design, conduct, and reporting of this work, and the evidence report on which this manuscript is based was reviewed by an external peer review panel (5). Methods Data Sources Data sources, including MEDLINE (from 1966), EMBASE (from 1980), HealthStar (from 1975), CancerLit (from 1983), and CINAHL (from 1983) were searched through October 1999 by using a strategy developed with a medical librarian (Table 1). Searches were limited to English-language studies in humans. We manually searched newly published relevant journal issues, bibliographies of included studies, and recent systematic reviews (6-9). To locate unpublished studies, we also contacted relevant professional societies and manufacturers of cytologic devices. Table 1. Search Strategy Study Selection We identified studies of conventional Pap testing (with or without manual rescreening), Pap testing using monolayer slide preparation (ThinPrep), Pap testing with primary computer screening (AutoPap or PAPNET), and Pap testing with computer rescreening (AutoPap or PAPNET). Other recently developed methods, the AutoCyte PREP System and the AutoCyte SCREEN system (TriPath Imaging, Inc., Burlington, North Carolina), had not been approved by the FDA at the time of our review and were not evaluated in this study. Study samples included women undergoing Pap testing for primary screening and those undergoing evaluation for previous cytologic abnormalities. The main outcome measures were the sensitivity and specificity of the cytologic test for detecting cases. Cytologic abnormality was defined by one of three thresholds: atypical squamous cells of undetermined significance (ASCUS), low-grade squamous intraepithelial lesions (LSIL), and high-grade squamous intraepithelial lesions (HSIL). Cases were defined as histologic diagnosis of cervical intraepithelial neoplasia (CIN), grades I to III, or carcinoma. Equivalent categories in other classification schemes (10-14) were also used (Figure). Figure. Map of classification schemes for cervical cytology. We included studies of the conventional Pap test if a reference standard of histologic examination or colposcopy was reasonably concurrent to the cytologic screening test (within 3 months) and if sufficient data were reported to complete all four cells of a 2 2 table. Comparison with such a reference standard provides a more relevant outcome for clinical decision makers because colposcopic or histologic diagnoses form the basis of most clinical management decisions. Only one study of ThinPrep (15) provided enough information to allow us to extract data on sensitivity and specificity compared with a gold standard of histologic examination or colposcopy. We therefore used a separate set of screening criteria for studies of the new methods, based on cytology society guidelines (16, 17) and FDA documents [18]: 1) The study must prospectively compare screening tests or test and reference standard on the same set of patients or slides; 2) if cytologic examination is the reference standard, discordant results from the two study tests must be adjudicated by an independent panel of experienced cytology professionals; 3) at least 50% of patients testing positive for HSIL must be verified by histologic examination or colposcopy; and 4) the study design must allow for separate analyses of sensitivity (or relative true-positive rate) and specificity (or relative false-positive rate). Data based on a cytologic reference standard cannot be integrated with data based on a histologic reference standard (19-23). However, when negative test results are not verified with the reference standard, information about incremental characteristics of test performance may be obtained by directly comparing independently applied conventional and new tests (21). In this case, both tests must be applied independently to all patients, and all positive results on either test must be verified with the reference standard. A relative true-positive rate and a relative false-positive rate, which can be used to determine relative estimates of the performance of the new test, can then be calculated. Two investigators independently screened each study. Differences of opinion were reconciled by consensus. The title and abstract of each citation were screened first, and the full report was screened second. Of the 1193 bibliographic references we reviewed, 761 (approximately 64%) were excluded on the basis of title or abstract. We reviewed the full reports of 346 studies of the conventional Pap test and 86 studies of the new methods (18 on AutoPap, 42 on PAPNET, and 26 on ThinPrep). We developed a numeric quality score to evaluate included citations. Nine members of the study's working group (6 clinicians, 2 economists, and 1 health policy analyst) initially identified more than 12 evaluation criteria on the basis of previously reported criteria (6, 24, 25). We used a consensus process to narrow this list to 7. Blinded to the rest of the group, each participant then independently assigned numerical weights to the criteria. The means of these votes were calculated. Each participant received a copy of his or her responses, depicted graphically in relation to the mean for each criterion, and was requested to confirm or