Public Reporting of Antibiotic Timing in Patients with Pneumonia: Lessons from a Flawed Performance Measure

Improving health care quality depends on having valid ways to measure quality. Unfortunately, there are few validated quality outcome measurements, because valid and feasible case-mix adjustors are lacking and patients are difficult to follow over time for clinically important outcomes, such as death. Processes of care are easier to identify and measure, but some of these measures will be proven invalid or inappropriate because their scientific rationale was flawed from the start, unanticipated consequences emerge after implementation, or later studies undermine them. We review how these issues played out in the measure of time to first antibiotic dose (TFAD), also called door-to-needle time, for patients presenting to the hospital with community-acquired pneumonia (CAP). We also propose lessons that can be learned from the experience. TFAD as a Quality Measure Community-acquired pneumonia is one of the most common admitting diagnoses in U.S. hospitals, accounting for more than 1 million hospitalizations yearly (1), with short-term mortality rates ranging from 0.5% to 27.1% (2). Given its risk, frequency, and perceived outcome variations, CAP was an obvious candidate for quality measurement and improvement initiatives. Because outcome measurement in CAP was problematic for the usual reasons (data collection burden, case-mix adjustment, and need for posthospital follow-up), investigators sought process measures associated with higher quality. During the 1990s, the notion of time-based quality measures gained favor because evidence emerged that rapid treatment of myocardial infarction, and later trauma, stroke, and sepsis, improved outcomes (37). Naturally, investigators began to examine whether rapid administration of antibiotics might improve CAP outcomes. In 1997, a retrospective study of 14069 Medicare patients hospitalized for CAP found that, after adjustment for severity (2) and demographic factors, administration of antibiotics within 8 hours was associated with a lower 30-day mortality rate (odds ratio [OR], 0.85 [95% CI, 0.75 to 0.96]) (8). Patients were included if they had chest radiography results within 2 days of admission consistent with pneumonia and an initial working diagnosis of pneumonia. In 2004, a second retrospective study of 13771 Medicare patients (age 65 years) hospitalized for CAP (9) also found that, among the 75% of patients without evidence of prehospital receipt of antibiotics, administration of antibiotics within 4 hours was associated with a lower 30-day mortality rate (OR, 0.85 [CI, 0.76 to 0.95]). Extrapolating these data to a hypothetical national Medicare sample, the authors estimated that achieving TFAD by 4 hours after presentation to the hospital would save more than 1200 lives yearly. The 2 studies reported that patients who received their first dose of antibiotics in the first hour of their emergency department stay had a higher mortality rate than those who received antibiotics later; however, this finding was attributed to incomplete adjustment for severity of CAP and was therefore not felt to challenge the main conclusion about TFAD (8, 9). Two smaller studies of CAP found no association between early antibiotic administration and outcomes (10, 11). Nevertheless, the authors of the 2004 study (9) editorialized that the 4-hour TFAD quality measure was still valid (12, 13). Translation into a Performance Standard Almost exclusively on the basis of results from the 1997 study, the Medicare National Pneumonia Project endorsed first antibiotics within 8 hours of hospital arrival as a CAP quality measure in 1998. The Medicare National Pneumonia Project tightened its TFAD window to 4 hours in 2002 on the basis of the prepublication results of the 2004 study by Houck and colleagues (9, 12). In 2003, the Infectious Diseases Society of America (IDSA) also endorsed a 4-hour timeframe (14). With support from the Medicare National Pneumonia Project and IDSA and subsequent endorsement by the National Quality Forum, The Joint Commission and The Centers for Medicare & Medicaid Services (CMS) chose the 4-hour TFAD measure as 1 of their initial core measures of quality (measure PN-5b). Since 2002, this measure has been publicly reported for all U.S. hospitals. In 2006, it became part of a measure set tied to additional payments under several pilot pay-for-performance programs (15). The Response from Emergency Medicine The emergency medicine community began raising red flags about the TFAD measure soon after its formulation, and complaints from this community markedly increased after TFAD was publicly reported and became the subject of pay-for-performance programs (16). Published studies challenging the measure soon followed. Although some questioned the association itself, most focused on the issue of diagnostic uncertainty. One study found that 22% of 86 randomly selected patients with pneumonia had uncertain presentations and often lacked infiltrates on chest radiography, which could have appropriately led to delayed antibiotic administration (17). Another study documented cases that were labeled poor-quality care, in which delayed use of antibiotics was clinically appropriate (18), whereas still another found that maneuvers to improve TFAD were not very cost-effective (19). In fact, many eligible patients with a working diagnosis of CAP who did not receive antibiotics within 4 hours had no radiographic evidence of pneumonia in the emergency department and did not have a final emergency department diagnosis of CAP (20, 21). Moreover, other studies showed that TFAD measurement led to administration of antibiotics in many patients who proved not to have pneumonia or another infectious disease (22, 23). Finally, a recent systematic review concluded that evidence from observational studies fails to confirm decreased mortality with early administration of antibiotics in stable patients with [CAP] (24). On the basis of these studies, analyses, and considerable anecdotal evidence, editorials in the emergency medicine literature argued vigorously for relaxing the TFAD standards (25, 26), pointing out that the measure was skewing emergency department triage priorities and promoting unnecessary antibiotic use (18). The Response from Payers, Regulators, and Professional Societies Within months of the critical publications, The Joint Commission and CMS revisited measure PN-5b. In October 2006, patients eligible for the measure had to have a final emergency department diagnosis of pneumonia (rather than an initial working diagnosis) and objective radiographic findings sometime during the hospitalization. Unfortunately, although the revised criteria solved some of the problems associated with PN-5b, they created new ones. For example, Fee and colleagues (27) worried that the new measures would generate pressure to administer antibiotics before patients were sent for computed tomography to rule out pulmonary embolism (even in the face of nondiagnostic chest radiographs) or to avoid writing pneumonia as the final emergency department diagnosis. In March 2007, IDSA and the American Thoracic Society issued joint guidelines that abolished time-specific goals for CAP treatment, now recommending that patients receive their first dose of antibiotics as soon as possible after a definitive diagnosis of CAP, preferably in the emergency department (28). One month later, The Joint Commission created a test measure (PN-5c) that relaxed the antibiotic administration window to 6 hours. That same month, the National Quality Forum withdrew its endorsement of measure PN-5b and endorsed PN-5c, which became the publicly reported measure in April 2008. In addition, The Joint Commission created a new data element, diagnostic uncertainty, which may exclude patients from TFAD measurement (29, 30). Whether all of these revisions will solve the problems associated with measure PN-5b is unknown; no study has yet shown a benefit from a 6-hour rule, and the diagnostic uncertainty construct has not, to our knowledge, been field-tested and validated. Unanticipated Consequences of TFAD Measurement and Reporting Prompt administration of antibiotics to patients with documented pneumonia makes sense, and seeking ironclad evidence to prove its value might seem to be analogous to requiring proof of the value of parachutes (31). Moreover, a randomized trial that withheld early antibiotic treatment in some patients with CAP would be unethical. It was therefore inevitable that decisions about the timing of antibiotic administration in CAP would be based on imperfect retrospective studies, out of necessity (8, 9). However, the TFAD measure was enacted largely on the evidence derived from 2 large studies, in which conditions (retrospective review of patients with working diagnoses of pneumonia) replicate only in part the predicament that busy emergency medicine physicians face daily: evaluating scores of patients with cough, fever, dyspnea, weakness, dizziness, confusion, or abdominal pain. As Pines (26) has written, Most ED [emergency department] patients do not present at triage with a sign on their forehead that reads, I have pneumonia; give me antibiotics now! Unlike myocardial infarction, in which there is palpable clinical urgency to confirm the diagnosis and a series of tests (cardiac biomarker measurement and electrocardiography) available to reliably do so, no gold standard test for pneumonia exists. Although a triage rule of obtaining an electrocardiogram in any patient whose symptoms, signs, or risk factors make myocardial infarction even a remote possibility makes perfect sense, a similar strategy for chest radiography would be resource intensive, often confusing (given the relatively poor sensitivity and specificity of the test in CAP [32]), impractical, and even potentially harmful (because of radiation exposure). In the days before measurement of TFAD, patients with uncertain diagnoses would continue to be evaluated until th