Clinical models can be improved by decreasing the importance assigned to fitting historical patient outcomes in often small and imperfectly characterized derivation cohorts. When Less Is More Clinical models play an important role in guiding patient care at the bedside, improving our understanding of diseases, and performing objective assessments of healthcare quality. The typical approach to developing these models places great importance on fitting historical patient outcomes in derivation data sets (such as those obtained from clinical studies or patient registries). However, for a fairly broad range of medical applications, these derivation data sets may be small after accounting for inclusionary and exclusionary criteria, and additionally may be imperfectly characterized due to noise and variations in the rates of patient outcomes. Collecting more data offers one approach to address this issue, but is challenging due to the costs and complexity of increasing the size of clinical cohorts. In the setting of small and imperfectly characterized data sets, approaches to developing clinical models that rely exclusively on fitting historical patient outcomes suffer from the implicit assumption that the derivation data sets are representative. Instead, as the new study by Chia et al. explores, the process of developing clinical models can be improved by decreasing the importance placed on fitting historical patient outcomes, and by supplementing these models with information about the extent to which patients differ from the statistical distribution of clinical characteristics within the derivation data set. When evaluated using data from three different clinical applications [patients with acute coronary syndrome enrolled in the DISPERSE2-TIMI33 and MERLIN-TIMI36 trials, patients undergoing inpatient surgery in the National Surgical Quality Improvement Program (NSQIP) registry, and patients undergoing percutaneous coronary intervention in the Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2) registry], this approach of treating derivation data for clinical models as simultaneously labeled and unlabeled consistently improved discrimination between high- and low-risk patients according to different statistical metrics. The idea of decreasing the importance assigned to fitting historical outcomes allows for better clinical models, and ultimately for improvements in the use of these models to study diseases, choose therapies, or evaluate healthcare providers. Conventional algorithms for modeling clinical events focus on characterizing the differences between patients with varying outcomes in historical data sets used for the model derivation. For many clinical conditions with low prevalence and where small data sets are available, this approach to developing models is challenging due to the limited number of positive (that is, event) examples available for model training. Here, we investigate how the approach of developing clinical models might be improved across three distinct patient populations (patients with acute coronary syndrome enrolled in the DISPERSE2-TIMI33 and MERLIN-TIMI36 trials, patients undergoing inpatient surgery in the National Surgical Quality Improvement Program registry, and patients undergoing percutaneous coronary intervention in the Blue Cross Blue Shield of Michigan Cardiovascular Consortium registry). For each of these cases, we supplement an incomplete characterization of patient outcomes in the derivation data set (uncensored view of the data) with an additional characterization of the extent to which patients differ from the statistical support of their clinical characteristics (censored view of the data). Our approach exploits the same training data within the derivation cohort in multiple ways to improve the accuracy of prediction. We position this approach within the context of traditional supervised (2-class) and unsupervised (1-class) learning methods and present a 1.5-class approach for clinical decision-making. We describe a 1.5-class support vector machine (SVM) classification algorithm that implements this approach, and report on its performance relative to logistic regression and 2-class SVM classification with cost-sensitive weighting and oversampling. The 1.5-class SVM algorithm improved prediction accuracy relative to other approaches and may have value in predicting clinical events both at the bedside and for risk-adjusted quality of care assessment.
[1]
Samprit Chatterjee,et al.
A Nonparametric Approach to Credit Screening
,
1970
.
[2]
E. DeLong,et al.
Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
,
1988,
Biometrics.
[3]
D. Hosmer,et al.
Applied Logistic Regression
,
1991
.
[4]
Vladimir Vapnik,et al.
Statistical learning theory
,
1998
.
[5]
Katharina Morik,et al.
Combining Statistical Learning with a Knowledge-Based Approach - A Case Study in Intensive Care Monitoring
,
1999,
ICML.
[6]
John Platt,et al.
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
,
1999
.
[7]
Bernhard Schölkopf,et al.
Estimating the Support of a High-Dimensional Distribution
,
2001,
Neural Computation.
[8]
V. L. Clark,et al.
The Blue Cross Blue Shield of Michigan Cardiovascular Consortium (BMC2) collaborative quality improvement initiative in percutaneous coronary interventions.
,
2002,
Journal of interventional cardiology.
[9]
Nitesh V. Chawla,et al.
SMOTE: Synthetic Minority Over-sampling Technique
,
2002,
J. Artif. Intell. Res..
[10]
Jeffrey S. Simonoff,et al.
Tree Induction Vs Logistic Regression: A Learning Curve Analysis
,
2001,
J. Mach. Learn. Res..
[11]
F. C. Dougherty,et al.
Predictors of 90-day outcome in patients stabilized after acute coronary syndromes.
,
2003,
European heart journal.
[12]
Nitesh V. Chawla,et al.
SPECIAL ISSUE ON LEARNING FROM IMBALANCED DATA SETS
,
2004
.
[13]
Nitesh V. Chawla,et al.
Editorial: special issue on learning from imbalanced data sets
,
2004,
SKDD.
[14]
S. Khuri,et al.
The NSQIP: a new frontier in surgery.
,
2005,
Surgery.
[15]
William Stafford Noble,et al.
Support vector machine
,
2013
.
[16]
Bernhard Schölkopf,et al.
Introduction to Semi-Supervised Learning
,
2006,
Semi-Supervised Learning.
[17]
Hsuan-Tien Lin,et al.
A note on Platt’s probabilistic outputs for support vector machines
,
2007,
Machine Learning.
[18]
C. Cannon,et al.
Safety, tolerability, and initial efficacy of AZD6140, the first reversible oral adenosine diphosphate receptor antagonist, compared with clopidogrel, in patients with non-ST-segment elevation acute coronary syndrome: primary results of the DISPERSE-2 trial.
,
2007,
Journal of the American College of Cardiology.
[19]
A. Skene,et al.
Effects of ranolazine on recurrent cardiovascular events in patients with non-ST-elevation acute coronary syndromes: the MERLIN-TIMI 36 randomized trial.
,
2007,
JAMA.
[20]
E. Braunwald,et al.
Effects of Ranolazine on Disease-Specific Health Status and Quality of Life Among Patients With Acute Coronary Syndromes: Results from the MERLIN-TIMI 36 Randomized Trial
,
2008,
Circulation. Cardiovascular quality and outcomes.
[21]
Filip De Turck,et al.
Bmc Medical Informatics and Decision Making Support Vector Machine versus Logistic Regression Modeling for Prediction of Hospital Mortality in Critically Ill Patients with Haematological Malignancies
,
2008
.
[22]
J. Birkmeyer,et al.
Prioritizing quality improvement in general surgery.
,
2008,
Journal of the American College of Surgeons.
[23]
M. Pencina,et al.
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond
,
2008,
Statistics in medicine.
[24]
Zeeshan Syed,et al.
Unsupervised Risk Stratification in Clinical Datasets: Identifying Patients at Risk of Rare Outcomes
,
2010,
ICML.