Development and validation of clinical prediction models: marginal differences between logistic regression, penalized maximum likelihood estimation, and genetic programming.

OBJECTIVE Many prediction models are developed by multivariable logistic regression. However, there are several alternative methods to develop prediction models. We compared the accuracy of a model that predicts the presence of deep venous thrombosis (DVT) when developed by four different methods. STUDY DESIGN AND SETTING We used the data of 2,086 primary care patients suspected of DVT, which included 21 candidate predictors. The cohort was split into a derivation set (1,668 patients, 329 with DVT) and a validation set (418 patients, 86 with DVT). Also, 100 cross-validations were conducted in the full cohort. The models were developed by logistic regression, logistic regression with shrinkage by bootstrapping techniques, logistic regression with shrinkage by penalized maximum likelihood estimation, and genetic programming. The accuracy of the models was tested by assessing discrimination and calibration. RESULTS There were only marginal differences in the discrimination and calibration of the models in the validation set and cross-validations. CONCLUSION The accuracy measures of the models developed by the four different methods were only slightly different, and the 95% confidence intervals were mostly overlapped. We have shown that models with good predictive accuracy are most likely developed by sensible modeling strategies rather than by complex development methods.

[1]  Diederick E Grobbee,et al.  Genetic programming outperformed multivariable logistic regression in diagnosing pulmonary embolism. , 2004, Journal of clinical epidemiology.

[2]  P. Royston,et al.  Stability of multivariable fractional polynomial models with selection of variables and transformations: a bootstrap investigation , 2003, Statistics in medicine.

[3]  J. Concato,et al.  A simulation study of the number of events per variable in logistic regression analysis. , 1996, Journal of clinical epidemiology.

[4]  Karel G M Moons,et al.  Ruling out deep venous thrombosis in primary care , 2005, Thrombosis and Haemostasis.

[5]  P. Austin.  A comparison of classification and regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality , 2007 .

[6]  J. Concato,et al.  The Risk of Determining Risk with Multivariable Models , 1993, Annals of Internal Medicine.

[7]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[8]  Ewout W Steyerberg,et al.  Internal and external validation of predictive models: a simulation study of bias and precision in small samples. , 2003, Journal of clinical epidemiology.

[9]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[10]  J. C. van Houwelingen,et al.  Predictive value of statistical models , 1990 .

[11]  Daniel B. Mark,et al.  TUTORIAL IN BIOSTATISTICS MULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS , 1996 .

[12]  Riccardo Poli,et al.  General Schema Theory for Genetic Programming with Subtree-Swapping Crossover: Part I , 2003, Evolutionary Computation.

[13]  E. Steyerberg.  Clinical Prediction Models , 2008, Statistics for Biology and Health.

[14]  Georgios Dounias,et al.  Evolving rule-based systems in two medical domains using genetic programming , 2004, Artif. Intell. Medicine.

[15]  A. Hoes,et al.  Excluding deep vein thrombosis safely in primary care. , 2006, The Journal of family practice.

[16]  K J Ottenbacher,et al.  Comparison of logistic regression and neural networks to predict rehospitalization in patients with stroke. , 2001, Journal of clinical epidemiology.

[17]  J. Hoak,et al.  Management of deep vein thrombosis and pulmonary embolism. A statement for healthcare professionals. Council on Thrombosis (in consultation with the Council on Cardiovascular Radiology), American Heart Association. , 1996, Circulation.

[18]  J. Habbema,et al.  Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. , 2001, Journal of clinical epidemiology.

[19]  M. Brezocnik,et al.  Predicting defibrillation success by 'genetic' programming in patients with out-of-hospital cardiac arrest. , 2003, Resuscitation.

[20]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[21]  H. Sox,et al.  Clinical prediction rules. Applications and methodological standards. , 1985, The New England journal of medicine.

[22]  Robert Gray,et al.  Flexible Methods for Analyzing Survival Data Using Splines, with Applications to Breast Cancer Prognosis , 1992 .

[24]  J. Habbema,et al.  Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. , 2000, Statistics in medicine.

[25]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[26]  S Forrest,et al.  Genetic algorithms , 1996, CSUR.

[27]  J. Hanley,et al.  A method of comparing the areas under receiver operating characteristic curves derived from the same cases. , 1983, Radiology.

[28]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[29]  P Royston,et al.  The use of fractional polynomials to model continuous risk variables in epidemiology. , 1999, International journal of epidemiology.

[30]  A Rogier T Donders,et al.  Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example. , 2004, Journal of clinical epidemiology.

[31]  J V Tu,et al.  Advantages and disadvantages of using artificial neural networks versus logistic regression for predicting medical outcomes. , 1996, Journal of clinical epidemiology.

[32]  R. D'Agostino,et al.  A comparison of performance of mathematical predictive methods for medical diagnosis: identifying acute cardiac ischemia among emergency department patients. , 1995, Journal of investigative medicine : the official publication of the American Federation for Clinical Research.

[33]  A. Laupacis,et al.  Clinical prediction rules. A review and suggested modifications of methodological standards. , 1997, JAMA.

[34]  Bradley Efron,et al.  Censored Data and the Bootstrap , 1981 .

[35]  P. J. Verweij,et al.  Penalized likelihood in Cox regression. , 1994, Statistics in medicine.

[36]  Patrick Royston,et al.  Simplifying a prognostic model: a simulation study based on clinical data , 2002, Statistics in medicine.

[37]  John R. Koza,et al.  Genetic Programming II , 1992 .

[38]  Geoffrey E. Hinton,et al.  A comparison of statistical learning methods on the Gusto database. , 1998, Statistics in medicine.

[39]  Nancy R. Cook,et al.  Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction , 2007, Circulation.

[40]  F. Harrell,et al.  Regression modelling strategies for improved prognostic prediction. , 1984, Statistics in medicine.

[41]  Peter C Austin,et al.  A comparison of regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality , 2007, Statistics in medicine.

[42]  P. Royston,et al.  A new approach to modelling interactions between treatment and continuous covariates in clinical trials by using fractional polynomials , 2004, Statistics in medicine.

[43]  William J. Long,et al.  Using Classification Tree and Logistic Regression Methods to Diagnose Myocardial Infarction , 1998, MedInfo.