Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions

Clinical prediction rules use clinical findings (history, physical examination, and test results) to make a diagnosis or predict an outcome (1-5). They quantify the relative importance of particular findings when evaluating an individual patient. How frequently these and other prediction rules are used in clinical practice is unknown, but they have become popular teaching aids in many training programs. This is not surprising, given that prediction rules were originally intended to help physicians interpret clinical information [and] know what clinical data are important to obtain (1). Recently, however, experts have equated prediction rules with decision rules, a subtle but important change. Laupacis and colleagues (2) proposed that the purpose of prediction rules is to suggest a diagnostic or therapeutic course of action. More ambitiously, the Evidence-Based Medicine Working Group (6) posited that prediction rules can change clinical behavior and reduce unnecessary costs while maintaining quality of care and patient satisfaction. This shift in purpose from predicting to decision making is notable for 2 reasons. First, very few prediction rules recommend decisions. Instead, most prediction rules provide diagnostic or prognostic probabilities, typically using a score (Table 1) or risk-stratification algorithm (Figure 1). Proponents of such ruleswhich intend to assist clinicians without telling them what to doassume that accurate predictions will improve clinical decisions. This assumption is questionable. For example, how the distinctions between low and moderate probabilities predicted by the rule in Figure 1 will (or should) affect clinicians' decisions is not obvious. Indeed, several studies suggest that access to accurate clinical predictions has an unpredictable effect on clinicians' decisions (7-10). Second, unlike those depicted in Figures 1 and 2 and Table 1, very few prediction rules have undergone formal impact analysis to determine whether they improve outcomes when used in clinical practice (6). In fact, few published prediction rules have described any clinical effects of their use (1, 2, 11). Thus, when using a prediction rule, clinicians usually do not know whether it will improve (or worsen) patient care. Table 1. Clinical Prediction Rule for Deep Venous Thrombosis Figure 1. Goldman and colleagues' clinical prediction rule for major cardiac complications for patients with chest pain. Figure 2. Ottawa Ankle Rule. Table 2 summarizes standards of evidence for developing and evaluating prediction rules, including the 4 levels of evidence proposed by the Evidence-Based Medicine Working Group (6, 12-47). We propose a fifth level of evidence because we believe that broad verification of a decision rule's clinical impact is no less important than that of the prediction rule on which it is based (48-51). These progressive evidentiary standards emphasize that a prediction rule rises to the level of a decision rule only if clinicians use its predictions to help make decisions for patients. This distinction is important because analyzing a decision rule's impact requires different methods and outcome measures than those needed to validate a prediction rule (52). For example, in advancing from a level 3 rule for predicting venous thrombosis (Table 1) to a level 5 decision rule, developers dichotomized the rule's predictions into 2 rather than 3 probability levels, further stratified these probabilities after bedside d-dimer testing, and then analyzed the impact of the rule's use on reducing unnecessary diagnostic imaging (outcome measure) (3, 48). These substantial differences between prediction rules and decision rules may partly explain why so few impact analyses have been performed (Table 2). Table 2. Developing and Evaluating Clinical Prediction Rules Accordingly, in our paper, we describe and illustrate the steps necessary to conduct an impact analysis of the prediction rule depicted in Figure 1 (52). The decision it addressesthe triage of patients in the emergency department with suspected acute cardiac ischemiaexemplifies occasions when decision rules are most likely to be useful: when decision making is complex, when the clinical stakes are high, or when there are opportunities to achieve cost savings without compromising patient care (6). We should note the differences between decision rules and other types of decision aids that are not discussed in our paper. Decision analysis quantifies the value of specified outcomes and uses data from the published literature to formulate health care policy (although it can be applied at the bedside as well). Most decision-support tools are designed to prevent errors when implementing decisions that are already made, whereas decision rules are designed to help clinicians make decisions. Practice guidelines address several issues in caring for patients with a particular syndrome, whereas decision rules address 1 discrete decision at 1 point in the continuum of care. Furthermore, when developed properly (Table 2), decision rules are exclusively evidence-based, their predictions are empirically validated, and their benefits are proven in clinical trials. In contrast, most practice guidelines reflect a consensus of expert opinion that is only sometimes supported by strong scientific evidence. Decision rules should not replace these other types of decision aids; rather, they can complement and strengthen them. Picking a Prediction Rule with Potential Impact In designing an impact analysis, the first step is to select the clinical prediction rule to use. For some clinical problems, such as triaging patients presenting to the emergency department with chest pain, several competing clinical prediction rules must be considered. Among the 5 previously published prediction rules relevant to this problem (4, 8, 53-55), 1 rule used only electrocardiographic predictors, which we did not find clinically sensible (54). We rejected 3 other rules because they predicted disease (the probability of acute myocardial infarction or unstable angina) rather than a full complement of health outcomes important to patients and physicians (including cardiac arrest, respiratory failure, and death) (8, 53, 55). Only Goldman and colleagues' rule (4) to predict complications of acute ischemic heart disease (Figure 1) aligned with our objective to reduce unnecessary admissions to inpatient monitored beds without increasing complications in patients triaged to less intensive care settings. Therefore, we investigated this prediction rule's validity, sensibility, and impact potential in our setting. Consider Validity The prediction rule (Figure 1) was derived in more than 10000 patients from 6 hospitals and was subsequently validated in nearly 5000 additional patients in 1 of those hospitals. These studies met level 2 evidentiary standards (Table 2) but also raised questions about the rule's predictive accuracy in the patient population served by large public hospitals. To address these questions, we used the prediction rule to risk-stratify consecutive cohorts of patients in our institution and found that both their distribution of risk strata and their risk-stratified complication rates were similar to those of Goldman and colleagues' validation cohort (4, 56, 57). These level 3 studies increased confidence in the predictive validity of the rule and also identified opportunities for improvement by showing that many patients admitted to our inpatient telemetry unit were patients at very low risk (according to the prediction rule) who had no complications (57). Consider Sensibility The evaluation of sensibility requires judgment, not statistical methods. Laupacis and colleagues (2) proposed that sensible prediction rules are those with logic that is clinically sensible and predictors that are both comprehensive (no potential predictors omitted from consideration) and appropriate for the purpose of the rule. Thus, input from clinicians who might use the prediction rule should be obtained when assessing its sensibility. Although our physicians were impressed by the rigor of the rule's development, they insisted that certain predictors were lacking: age, risk factors for atherosclerosis, results of previous diagnostic testing (coronary angiography), previous treatment (coronary revascularization), and additional electrocardiographic criteria for myocardial ischemia or infarction (new left bundle-branch block). Consider Impact Potential In their validation study of the prediction rule, Goldman and colleagues (4) demonstrated that the prediction rule's specificity was far superior to that of physicians' actual (unaided) decisions (57% vs. 39%; P< 0.001), meaning that the prediction rule outperformed the physicians in correctly identifying patients who will not have complications. However, the rule's sensitivity was slightly inferior to that of physicians' unaided decisions (91% vs. 96%; P= 0.04). This suggested that the prediction rule could reduce unnecessary use of inpatient monitored beds but with an important (and perhaps unacceptable) tradeoff: more patients having major cardiac complications in unmonitored locations. To explore this issue further, we performed a simulated impact analysis of the prediction rule, using 20 written cases presented to 147 physicians at our institution (58). This exercise showed that the prediction rule's predictions were more sensitive and more specific than our physicians' (simulated) decisions. In addition, we documented marked variation in decision making by our physicians, none of whom equaled the prediction rule in both sensitivity and specificity. Thus, we hypothesized that our physicians, by using the prediction rule, might improve their specificity without reducing their sensitivity in correctly identifying patients who would have complications. If so, use of the prediction rule would achieve the impact that we sought. Preparing for Impact Analysis After iden