The Effect of Clustering of Outcomes on the Association of Procedure Volume and Surgical Outcomes

Context While studies suggest that surgical patients fare best with providers (hospitals and surgeons) that perform a high volume of procedures, most have not accounted for the tendency of patients of 1 provider to have similar outcomes (clustering). When outcomes of individual patients are clustered, more patients are required to prove that providers' outcomes differ from one another. Contribution This reanalysis of data from 3 previously published volumeoutcome studies of surgery accounted for clustering within provider. Statistical significance of volumeoutcome relationships of morbidity end points was attenuated substantially after adjustment for the effects of clustering. Implications Planners considering regionalizing surgery should remember that volumeoutcome studies that have not accounted for clustering exaggerate differences in outcomes by provider. The Editors In an attempt to evaluate the degree to which the choice of provider affects outcomes of major medical procedures, numerous investigators have used procedure volume as a proxy for expertise and have conducted studies correlating volume with outcomes. However, volume is a crude, easily calculated measure, and its use may overlook large variations in quality among providers that are independent of the number of procedures performed. If there are such large variations in outcomes among providers, the outcomes of patients treated by the same provider are necessarily correlated, or clustered. Clustering of this nature invalidates conventional statistical analyses. Patients treated at the same hospital or by the same surgeon may be more likely to experience similar outcomes if surgical technique or supportive care practices vary among providers and if these factors affect outcomes. It is well established that statistical methods must correct for the effect of clustering of this nature if it exists (1-3). In general, correction for clustering attenuates the statistical significance of observed trends. In this article, we describe measures that characterize the degree of clustering of outcomes and methods that permit the calculation of tests of statistical significance that are adjusted for the impact of clustering. We reanalyzed 3 large studies of major cancer surgery (4-7) and found examples that illustrate varying degrees of clustering. This reanalysis focused on the relationship of surgeon volume to outcome. We explored the effect of correction for clustering on the statistical significance of tests for association between surgeon volume and outcome. Graphical methods that visually characterize the degree of between-surgeon variation in quality after adjustment for volume and case mix are described. We also explored analyses that attempt to distinguish between the effects of surgeon volume and those of hospital volume. Methods Data Sources All 3 data sets were obtained by using the linkage of the Surveillance, Epidemiology, and End Results registries with Medicare claims from the Centers for Medicare & Medicaid Services (formerly Health Care Financing Administration) (8). These linkages permit evaluation of the outcomes of surgery for all patients who received a diagnosis of cancer in the geographical regions defined by the registries. Patients were enrolled in Part A and Part B Medicare but were not enrolled in a health maintenance organization, and each had a diagnosis of colon, prostate, or rectal cancer made at age 65 years or older in a region covered by one of the Surveillance, Epidemiology, and End Results registries. All tumors were primary, invasive, and malignant and were diagnosed during 1992 to 1996. The sample sizes for the 3 cohorts ranged from very large (colon cancer, n = 24166) to large (prostate cancer, n = 10737) to moderate (rectal cancer, n = 2603). These analyses encompassed 2682, 999, and 1141 surgeons, respectively. The median numbers of patients per surgeon were 5 (range, 1 to 85), 7 (range, 1 to 121), and 1 (range, 1 to 26), respectively. The particular billing codes used to identify the surgeon, the specific surgical procedures, and the outcomes have been described in detail in the original publications (4-7), but our essential approach is summarized as follows. Procedure Volume Surgeon volume was defined as the total number of operations performed by the given surgeon between 1992 and 1996 on members of each cohort, as ascertained from the Medicare files on the basis of International Classification of Diseases, Ninth Revision, procedure codes (9). The Common Procedural Terminology codes submitted in the National Claims History files were used to identify the procedures (10). Surgeon identity was established on the basis of a unique identifier that has been mandatory on all claims since 1991. Outcomes Patient outcomes were binary indicators for 2-year mortality or a particular postoperative complication, depending on the cancer cohort. For the colon and rectal cancer data sets, mortality at 2 years and occurrence of a procedure impairing fecal continence (ileostomy or colostomy in colon cancer and an abdominoperineal resection in rectal cancer) were the outcomes evaluated. These measures of morbidity were selected because they can have an important impact on the patient's subsequent quality of life. For the prostate cancer cohort, postoperative and late urinary complications were examined. Postoperative complications were defined as potentially life-threatening cardiac, respiratory, or vascular events; the need for reoperation; and hemorrhage, renal failure, or shock, all occurring within 30 days of surgery. Late urinary complications were identified by procedures or symptoms recorded more than 30 days after but within 1 year of surgery and included diagnoses of bladder-neck obstruction, urethral or ureteral strictures, intestinal or vesical fistulas, pelvic abscess, and other urinary tract complications that required surgical repair. Case Mix In each primary analysis, clinical and demographic variables were used to adjust for differences in patient case mix among individual surgeons. These included age; sex; ethnicity; disease stage at diagnosis; disease grade; income level; and the presence of particular features such as emergent presentation, obstruction, or perforation. All results shown in the tables and figures are adjusted for case mix. Alternative Statistical Methods for Cluster-Correlated Data We used and compared 2 general statistical approaches that permit adjustments to account for the effect of clustering of outcomes on statistical significance levels: random-effects models, in which the impact of each surgeon is modeled explicitly (11), and generalized estimating equations (GEEs), in which the underlying patient-specific analysis is adjusted to accommodate the effect of clustering on the statistical significance of the analyses (12). The GEE method does not alter the estimate of the magnitude of the impact of surgeon volume on outcome; only the statistical significance (and the width of the confidence interval) is altered. In the random-effects model, the estimate may change after correction for clustering. Both of these methods are widely used in health services research and clinical research. We evaluated the effect of surgeon volume on outcomes in 3 settings: adjustment solely for case mix; adjustment for case mix and hospital volume; and adjustment for case mix, hospital volume, and clustering. Logistic regression was used throughout to estimate the impact of volume on outcome. The random-effects models were fitted by using the gllamm6 command in Stata, version 7.0 (Stata Corp., College Station, Texas), and the GEE models were fitted by using the GENMOD procedure in SAS, version 8.0 (SAS Institute, Inc., Cary, North Carolina). Volume was modeled as a continuous variable in all analyses. Odds ratios are reported per 100-unit decline in volume increments for the colon and prostate cancer analyses and per 10-unit decline for the rectal cancer analysis. A Graphical Assessment and Statistical Test for Clustering of Outcomes We used a graphical technique to characterize visually the extent of variation among providers. Only surgeons with a minimum number of patients treated were used to ensure that each surgeon had a sufficient number of patients in his or her profile to allow meaningful estimates of the mean departure from the overall event rate. As a result, we did not construct graphs for the rectal cancer data set, since the vast majority of the surgeons treated very few patients. The observed outcome relative frequencies, that is, the proportion of patients who experienced the outcome, were evaluated for each surgeon. A conventional multivariable logistic regression analysis that included surgeon volume, hospital volume, and case mix as predictor variables was used to estimate the patient-specific outcome probabilities. The sum of these probabilities for each patient in the surgeon's profile was calculated as the expected event rate for the surgeon. To obtain the graph, the observed frequencies were plotted in a histogram. An expected histogram was created to display the outcome histogram that would be expected if there was no additional surgeon-to-surgeon variation after adjustment for known predictors of outcome in the model. For each surgeon, the binomial distribution based on the preceding expected event rate and the number of patients in the surgeon's profile was used to determine probabilities for each possible observed outcome relative frequency (in a histogram grouped by cells of 5 percentage points). For each cell of the histogram, these probabilities were summed across all surgeons to obtain the expected histogram. This expected histogram reflects the degree of spread we would expect to observe in the absence of clustering. The spread of the observed distribution will exceed the spread of the expected distribution in the presence of clustering. To test whether differences between the 2 histogr