Assessing the Generalizability of Prognostic Information

Your secretary just called to say that a favorite patient of yours (a 45-year-old high school teacher) with colon cancer dropped off his surgical and pathologic reports. She reminds you that he is scheduled this afternoon for a second opinion about his prognosis. Scanning the reports, you note that the surgeon staged the cancer at Dukes stage C1 on the basis of negative margins and 1 positive out of 28 lymph nodes and that the pathologist staged the microscopic tissue at Jass stage IV. You are not sure how to translate these stages into useful prognostic information for your patient. Time is short. You log on to MEDLINE, type Dukes and Jass, and select the years 1966 to 1997. The combined terms generate 18 references (1-18). Of these, 4 are independent reports of observed mortality rates (8, 11, 17, 19). You also track down the original reports of the Dukes and Jass systems (20, 21) and learn that both systems were developed at St. Mark's Hospital in London more than 30 years ago (20, 21). The Dukes system is based on histology and extent of local, lymphatic, and venous spread seen in the surgical specimen (20), and the Jass system is based on microscopic pathologic staging (21). However, the reported mortality rates by stage for these systems vary widely. How do you tell which report and which system are most likely to pertain to a 45-year-old teacher from Cleveland, Ohio? Physicians are frequently asked for prognostic assessments and often worry that their assessments will prove inaccurate (22, 23). Prognostic systems, including risk factors, staging systems, decision rules, statistical models, and computer algorithms, have been developed to standardize and enhance the accuracy of prognostic assessments (24, 25). Although diverse techniques are used to develop these systems, all use a sample of patients for whom the outcome is known to relate baseline characteristics to an outcome of interest. Once a system is developed, it can be used to generate predictions for patients whose outcome is not yet known. A common problem in the application of prognostic systems is that the accuracy of the predictions degrades from the sample in which the system was first developed to subsequent application; that is, the systems do not generalize (26). Although much has been written on the evaluation and reporting of prognostic systems (25, 27-40), few investigations have directly addressed the issue of generalizability, also known as external validity (29). Instead, discussion has focused on evaluating issues of internal validity, such as the sample in which the system was developed, the variables used in the model, the techniques used in system development, or the accuracy of the system in the sample in which it was developed. These factors offer important insights into the probable generalizability of the system to a new sample of patients, but they do not directly test subsequent performance (25). We discuss the importance of systematically testing the subsequent performance of a system. We begin by defining the relation between accuracy and generalizability, components of accuracy (calibration and discrimination), and components of generalizability (reproducibility and transportability). We then discuss issues of transportability and propose a five-level hierarchy of external validity based on the type and degree of transportability tested. We illustrate this approach with a structured review of the Dukes and Jass staging systems for colon and rectal cancer as applied to the 45-year-old teacher described previously. Because this approach treats prognostic system development as a black box and focuses on subsequent performance, it can be applied to any prognostic system, no matter how complex. Accuracy and Generalizability Accuracy and generalizability are related concepts (Table 1). Accuracy is the degree to which predictions match outcomes. Generalizability is the ability of the system to provide accurate predictions in a different sample of patients. Table 1. Definitions of Accuracy and Generalizability Components of Accuracy A series of numeric predictions may be inaccurate in two ways. The predicted probability may be too high or too low (an error in calibration), or the relative ranking of individual risk may be out of order (an error in discrimination). Assume that we have observed a sample of patients with colon cancer for whom the overall 5-year mortality rate was 50%. A system that predicted a 50% probability of death at 5 years for each patient would be perfectly calibrated. However, it would not discriminate among patients who lived and those who died within the interval. Conversely, a system that assigned a 10% probability of death at 5 years to patients who lived and an 11% probability to those who died would be perfectly discriminating but poorly calibrated. The relative importance of calibration and discrimination depends on the intended application. If predictions are used to counsel a patient, the accuracy of the numeric probability (calibration) is important. Patients are not concerned about how sick they are relative to other patients with the disease; instead, they are concerned with the likelihood that their disease will result in death or some other important outcome (such as, in the case of our hypothetical patient, the inability to handle the challenges of teaching) within a defined period of time. Calibration is also important in health services research. When predicted and observed mortality rates are compared to identify unexpectedly high or low rates, errors in calibration can cause large numbers of hospitals or providers to appear to have excessively high or low rates of mortality when, in fact, the model is not calibrated (41). In contrast, if predictions are used to stratify patients by stage of severity in order to compare treatments within a given stage, the important aspect of accuracy is whether patients whose disease is within a stage are equally likely to experience the outcome and that the stages are correctly ranked in order of risk (discrimination) (41). Calibration and discrimination are evaluated in different ways. Calibration is not routinely measured but can be illustrated by using calibration curves, which plot predicted versus observed outcomes (42). Discrimination is commonly measured by using the area under the receiver-operating characteristic (ROC) curve (43). This area ranges from 0.5 (no discrimination) to 1.0 (perfect discrimination) and reflects the probability that in all possible pairs of patients in which one patient lives and one dies, a higher risk is assigned to the patient who died than to the one who lived. The area under the ROC curve can be directly calculated from a table of observed and predicted outcomes (44). It can also be calculated for continuous prognostic estimates (with or without censored observations) by using the C statistic (37). Several sources provide more thorough discussion of measures of calibration and discrimination (37, 41-43, 45-49). Components of Generalizability No matter how calibrated and discriminating a system may be in development, a system that can only predict outcomes in the sample in which it was developed is useless (25, 50). For a system to be generalizable, the accuracy (that is, calibration and discrimination) of the system must be both reproducible and transportable. Reproducibility Reproducibility requires the system to replicate its accuracy in patients who were not included in development of the system but who are from the same underlying population. A test of the reproducibility of the system evaluates the degree to which the system is fit to real patterns in the data rather than to random noise. The system is more likely to be fit to random noise (overfit) when the ratio of the number of variables to the number of patients experiencing events is small (37). If a prognostic system is overfit, it may not generalize well. Methods for evaluating the reproducibility of a prognostic system have been thoroughly described elsewhere (32, 35, 51, 52) and are based on the use of data resampling techniques (such as bootstrapping) to evaluate the degree of overfitting. Bootstrapping techniques can evaluate errors in discrimination and in calibration and are particularly important when the sample used to develop the model is small (35, 52). Transportability Alternatively, a system may be reproducible (that is, perform well when tested by bootstrapping) and degrade in subsequent patient samples because of underfitting (37, 53). Underfitting occurs when important independent predictors of outcome are omitted from the system. For example, a system for breast cancer that omits the presence of metastasis may perform well in a sample of patients with no metastatic disease and degrade badly when it is tested in a more diverse sample. Bootstrapping of the development sample would not detect this problem because the development sample is homogeneous with respect to metastatic disease. Omission of metastasis in breast cancer staging is, of course, an obvious mistake; however, not all important prognostic variables in a disease state are known. Because any given sample may be homogeneous for an important variable (known or unknown), it is important to test the system in subsequent samples. Transportability requires the system to produce accurate predictions in a sample drawn from a different but plausibly related population or in data collected by using slightly different methods than those used in the development sample. In the case of our patient, we want to know whether staging systems developed at St. Mark's Hospital in London more than 30 years ago pertain to a 45-year-old teacher from Cleveland whose disease was staged at Case Western Reserve University Hospital in 1998. Systems that are underfit may demonstrate reproducibility but not consistent transportability (50). Therefore, data from a separate, nonidentical sample are needed t