Optimizing ACS NSQIP modeling for evaluation of surgical quality and risk: patient risk adjustment, procedure mix adjustment, shrinkage adjustment, and surgical focus.

The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) collects detailed clinical data from participating hospitals using standardized data definitions, analyzes these data, and provides participating hospitals with reports that permit risk-adjusted comparisons with a surgical quality standard. Since its inception, the ACS NSQIP has worked to refine surgical outcomes measurements and enhance statistical methods to improve the reliability and validity of this hospital profiling. From an original focus on controlling for between-hospital differences in patient risk factors with logistic regression, ACS NSQIP has added a variable to better adjust for the complexity and risk profile of surgical procedures (procedure mix adjustment) and stabilized estimates derived from small samples by using a hierarchical model with shrinkage adjustment. New models have been developed focusing on specific surgical procedures (eg, "Procedure Targeted" models), which provide opportunities to incorporate indication and other procedure-specific variables and outcomes to improve risk adjustment. In addition, comparative benchmark reports given to participating hospitals have been expanded considerably to allow more detailed evaluations of performance. Finally, procedures have been developed to estimate surgical risk for individual patients. This article describes the development of, and justification for, these new statistical methods and reporting strategies in ACS NSQIP.

[1]  Karl Hammermeister,et al.  Successful Implementation of the Department of Veterans Affairs’ National Surgical Quality Improvement Program in the Private Sector: The Patient Safety in Surgery Study , 2008, Annals of surgery.

[2]  S L Normand,et al.  Cardiac surgery report cards: comprehensive review and statistical critique. , 2001, The Annals of thoracic surgery.

[3]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[4]  Sharon-Lise T. Normand,et al.  Statistical and Clinical Aspects of Hospital Outcomes Profiling , 2007, 0710.4622.

[5]  K. Hammermeister,et al.  Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. , 1998, Journal of the American College of Surgeons.

[6]  F. Grover,et al.  Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. , 1997, Journal of the American College of Surgeons.

[7]  Paul D. Allison,et al.  Logistic regression using sas®: theory and application , 1999 .

[8]  J. Zimmerman,et al.  Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited* , 2007, Critical care medicine.

[9]  E. Hannan,et al.  Predicting risk-adjusted mortality for trauma patients: logistic versus multilevel logistic models. , 2010, Journal of the American College of Surgeons.

[10]  Simon Jackman,et al.  Bayesian Analysis for the Social Sciences , 2009 .

[11]  Gabriel J. Escobar,et al.  Effect of Choice of Estimation Method on Inter-Hospital Mortality Rate Comparisons , 2010, Medical care.

[12]  Tianxi Cai,et al.  The Performance of Risk Prediction Models , 2008, Biometrical journal. Biometrische Zeitschrift.

[13]  C. Ko,et al.  Does Surgical Quality Improve in the American College of Surgeons National Surgical Quality Improvement Program: An Evaluation of All Participating Hospitals , 2009, Annals of surgery.

[14]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[15]  C. Ko,et al.  Comparison of Outlier Identification Methods in Hospital Surgical Quality Improvement Programs , 2010, Journal of Gastrointestinal Surgery.

[16]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[17]  B. Efron,et al.  Stein's Paradox in Statistics , 1977 .

[18]  Using a hierarchical model to estimate risk-adjusted mortality for hospitals not included in the reference sample. , 2010, Health services research.

[19]  E. Livingston,et al.  Tread carefully with stepwise regression. , 2010, Archives of surgery.

[20]  C. Ko,et al.  Improving American College of Surgeons National Surgical Quality Improvement Program risk adjustment: incorporation of a novel procedure risk score. , 2010, Journal of the American College of Surgeons.

[21]  E. DeLong,et al.  Comparing risk-adjustment methods for provider profiling. , 1998, Statistics in medicine.

[22]  C. Ko,et al.  Reliability Adjustment for Reporting Hospital Outcomes With Surgery , 2012, Annals of surgery.

[23]  Justin B Dimick,et al.  Risk adjustment for comparing hospital quality with surgery: how many variables are needed? , 2010, Journal of the American College of Surgeons.

[24]  S. F. Buck.  A Method of Estimation of Missing Values in Multivariate Data Suitable for Use with an Electronic Computer , 1960 .

[25]  William Stafford Noble,et al.  Support vector machine , 2013 .

[26]  Johan Hopstadius,et al.  Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery , 2011, Statistical methods in medical research.

[27]  Harvey Goldstein,et al.  Multilevel modelling of health statistics , 2001 .

[28]  T. Osler,et al.  Measuring quality for public reporting of health provider quality: making it meaningful to patients. , 2010, American journal of public health.

[29]  C. Ko,et al.  Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling. , 2009, Journal of the American College of Surgeons.

[30]  S Greenland,et al.  Principles of multilevel modelling. , 2000, International journal of epidemiology.

[31]  C. Ko,et al.  The impact of adjusting for reliability on hospital quality rankings in vascular surgery. , 2011, Journal of vascular surgery.

[32]  K Ulm,et al.  A simple method to calculate the confidence interval of a standardized mortality ratio (SMR) , 1990, American journal of epidemiology.

[33]  Helen Brown,et al.  Applied Mixed Models in Medicine , 2000, Technometrics.

[34]  Peter C. Austin,et al.  The Relationship Between the C-Statistic of a Risk-adjustment Model and the Accuracy of Hospital Report Cards: A Monte Carlo Study , 2013, Medical care.

[35]  Lynne Moore,et al.  Evaluating the performance of trauma centers: hierarchical modeling should be used. , 2010, The Journal of trauma.

[36]  J. Birkmeyer,et al.  Ranking hospitals on surgical mortality: the importance of reliability adjustment. , 2010, Health services research.

[37]  Filip De Turck,et al.  Bmc Medical Informatics and Decision Making Support Vector Machine versus Logistic Regression Modeling for Prediction of Hospital Mortality in Critically Ill Patients with Haematological Malignancies , 2008 .

[38]  B. Hamilton,et al.  The Impact of Surgeon Specialization on Patient Mortality: Examination of a Continuous Herfindahl-Hirschman Index , 2009, Annals of surgery.

[39]  Mitchell A H Levine,et al.  Relative risk and odds ratio data are still portrayed with inappropriate scales in the medical literature. , 2010, Journal of clinical epidemiology.

[40]  C L Christiansen,et al.  Improving the Statistical Approach to Health Care Provider Profiling , 1997, Annals of Internal Medicine.

[41]  C. Ko,et al.  Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. , 2009, Journal of the American College of Surgeons.

[42]  T. Sundt,et al.  Massachusetts Cardiac Surgery Report Card: Implications of Statistical Methodology , 2007 .

[43]  E. DeLong.  Hierarchical modeling: its time has come. , 2003, American heart journal.

[44]  N. Cook.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[45]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[46]  W. Chow,et al.  Relevance of the c-statistic when evaluating risk-adjustment models in surgery. , 2012, Journal of the American College of Surgeons.