A comparative analysis of predictive models of morbidity in intensive care unit after cardiac surgery – Part I: model planning

BackgroundDifferent methods have recently been proposed for predicting morbidity in intensive care units (ICU). The aim of the present study was to critically review a number of approaches for developing models capable of estimating the probability of morbidity in ICU after heart surgery. The study is divided into two parts. In this first part, popular models used to estimate the probability of class membership are grouped into distinct categories according to their underlying mathematical principles. Modelling techniques and intrinsic strengths and weaknesses of each model are analysed and discussed from a theoretical point of view, in consideration of clinical applications.MethodsModels based on Bayes rule, k- nearest neighbour algorithm, logistic regression, scoring systems and artificial neural networks are investigated. Key issues for model design are described. The mathematical treatment of some aspects of model structure is also included for readers interested in developing models, though a full understanding of mathematical relationships is not necessary if the reader is only interested in perceiving the practical meaning of model assumptions, weaknesses and strengths from a user point of view.ResultsScoring systems are very attractive due to their simplicity of use, although this may undermine their predictive capacity. Logistic regression models are trustworthy tools, although they suffer from the principal limitations of most regression procedures. Bayesian models seem to be a good compromise between complexity and predictive performance, but model recalibration is generally necessary. k-nearest neighbour may be a valid non parametric technique, though computational cost and the need for large data storage are major weaknesses of this approach. Artificial neural networks have intrinsic advantages with respect to common statistical models, though the training process may be problematical.ConclusionKnowledge of model assumptions and the theoretical strengths and weaknesses of different approaches are fundamental for designing models for estimating the probability of morbidity after heart surgery. However, a rational choice also requires evaluation and comparison of actual performances of locally-developed competitive models in the clinical scenario to obtain satisfactory agreement between local needs and model response. In the second part of this study the above predictive models will therefore be tested on real data acquired in a specialized ICU.

[1]  Paolo Barbini,et al.  A comparative analysis of predictive models of morbidity in intensive care unit after cardiac surgery – Part II: an illustrative example , 2007, BMC Medical Informatics Decis. Mak..

[2]  N. Obuchowski,et al.  ROC curves in clinical chemistry: uses, misuses, and possible solutions. , 2004, Clinical chemistry.

[3]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[4]  Joseph Eliahoo,et al.  Cardiac surgery risk modeling for mortality: a review of current practice and suggestions for improvement. , 2004, The Annals of thoracic surgery.

[5]  Gary D. Friedman Primer of epidemiology , 1980 .

[6]  J. Wennberg,et al.  Multivariate Prediction of In‐Hospital Mortality Associated With Coronary Artery Bypass Graft Surgery , 1992 .

[7]  C A Bodian,et al.  Intraoperative hemodynamic predictors of mortality, stroke, and myocardial infarction after coronary artery bypass surgery. , 1999, Anesthesia and analgesia.

[8]  G. Beck,et al.  Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients. A clinical severity score. , 1992, JAMA.

[9]  K. Wegscheider,et al.  Outcome prediction models on admission in a medical intensive care unit: do they predict individual outcome? , 1990, Critical care medicine.

[10]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[11]  G A Diamond,et al.  What price perfection? Calibration and discrimination of clinical prediction models. , 1992, Journal of clinical epidemiology.

[12]  P. J. Green,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[13]  E. DeLong,et al.  Comparing risk-adjustment methods for provider profiling. , 1998, Statistics in medicine.

[14]  G. Beck,et al.  ICU admission score for predicting morbidity and mortality risk after coronary artery bypass grafting. , 1997, The Annals of thoracic surgery.

[15]  Alexander Tropsha,et al.  k Nearest Neighbors QSAR Modeling as a Variational Problem: Theory and Applications , 2005, J. Chem. Inf. Model..

[16]  B. McNeil,et al.  Predicting Mortality after Coronary Artery Bypass Surgery , 1998, Medical decision making : an international journal of the Society for Medical Decision Making.

[17]  Balaji Rajagopalan,et al.  Statistical downscaling using K‐nearest neighbors , 2005 .

[18]  S. Lemeshow,et al.  Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients. , 1993, JAMA.

[19]  James R Carpenter,et al.  Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom* , 2006, Critical care medicine.

[20]  Wojtek J. Krzanowski,et al.  Principles of multivariate analysis : a user's perspective. oxford , 1988 .

[21]  N. Graham,et al.  Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation , 2002 .

[22]  David G. Stork,et al.  Pattern Classification , 1973 .

[23]  M. Newman,et al.  Intraoperative physiologic variables and outcome in cardiac surgery: Part I. In-hospital mortality. , 2000, Annals of Thoracic Surgery.

[24]  W. Knaus The APACHE III Prognostic System , 1992 .

[25]  N J Starr,et al.  Predictors of outcome in cardiac surgical patients with prolonged intensive care stay. , 1997, Chest.

[26]  A L Shroyer,et al.  Bayesian-logit model for risk assessment in coronary artery bypass grafting. , 1994, The Annals of thoracic surgery.

[27]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[28]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[29]  John Skilling,et al.  Data analysis : a Bayesian tutorial , 1996 .

[30]  Bernard W. Silverman,et al.  Density Estimation for Statistics and Data Analysis , 1987 .

[31]  Torgny Groth,et al.  Methods for selection of adequate neural network structures with application to early assessment of chest pain patients by biochemical monitoring , 2000, Int. J. Medical Informatics.

[32]  A. Azzalini Logistic regression for autocorrelated data with application to repeated measures , 1994 .

[33]  Anthony Ralston,et al.  Statistical Methods for Digital Computers. , 1980 .

[34]  Anil K. Jain,et al.  39 Dimensionality and sample size considerations in pattern recognition practice , 1982, Classification, Pattern Recognition and Reduction of Dimensionality.

[35]  Shigeo Abe DrEng Pattern Classification , 2001, Springer London.

[36]  Diego A. Alvarez,et al.  Comparison between logistic regression and neural networks to predict death in patients with suspected sepsis in the emergency room , 2005, Critical care.

[37]  H. Krumholz Mathematical models and the assessment of performance in cardiology. , 1999, Circulation.

[38]  J. C. van Houwelingen,et al.  Logistic Regression for Correlated Binary Data , 1994 .

[39]  P. Barbini,et al.  A multivariate Bayesian model for assessing morbidity after coronary artery surgery , 2006, Critical care.

[40]  Lucila Ohno-Machado,et al.  The use of receiver operating characteristic curves in biomedical informatics , 2005, J. Biomed. Informatics.

[41]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[42]  B Bridgewater,et al.  Predicting operative risk for coronary artery surgery in the United Kingdom: a comparison of various risk prediction algorithms , 1998, Heart.

[43]  James R. Schott,et al.  Principles of Multivariate Analysis: A User's Perspective , 2002 .

[44]  D. Hosmer,et al.  Applied Logistic Regression , 1991 .

[45]  Constantin F. Aliferis,et al.  A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis , 2004, Bioinform..

[46]  F H Edwards,et al.  1988: use of a Bayesian statistical model for risk assessment in coronary artery surgery. Updated in 1995. , 1995, The Annals of thoracic surgery.

[47]  M. Hippeläinen,et al.  Intra-institutional prediction of outcome after cardiac surgery: comparison between a locally derived model and the EuroSCORE. , 2000, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[48]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[49]  A Agresti,et al.  On Logit Confidence Intervals for the Odds Ratio with Small Samples , 1999, Biometrics.

[50]  J L Moran,et al.  Mortality and other event rates: what do they tell us about performance? , 2003, Critical care and resuscitation : journal of the Australasian Academy of Critical Care Medicine.

[51]  F. Grover,et al.  Cardiac surgery risk models: a position article. , 2004, The Annals of thoracic surgery.

[52]  P. Schulman Bayes' theorem--a review. , 1984, Cardiology clinics.

[53]  Nicolette F de Keizer,et al.  Performance of prognostic models in critically ill cancer patients – a review , 2005, Critical care.

[54]  M. Amrani,et al.  An evaluation of existing risk stratification models as a tool for comparison of surgical performances for coronary artery bypass grafting between institutions. , 2003, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[55]  M. Newman,et al.  Intraoperative physiologic variables and outcome in cardiac surgery: Part II. Neurologic outcome. , 2000, The Annals of thoracic surgery.

[56]  S Lemeshow,et al.  Effect of changing patient mix on the performance of an intensive care unit severity-of-illness model: how to distinguish a general from a specialty intensive care unit. , 1996, Critical care medicine.

[57]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[58]  S L Zeger,et al.  Regression analysis for correlated data. , 1993, Annual review of public health.

[59]  M W Knuiman,et al.  An empirical comparison of multivariable methods for estimating risk of death from coronary heart disease. , 1997, Journal of cardiovascular risk.

[60]  E. Draper,et al.  APACHE II: A severity of disease classification system , 1985, Critical care medicine.

[61]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[62]  G. Avanzolini,et al.  Classification of postoperative cardiac patients: comparative evaluation of four algorithms. , 1991, International journal of bio-medical computing.

[63]  J. Marshall,et al.  Should morbidity replace mortality as an endpoint for clinical trials in intensive care? , 1995, The Lancet.

[64]  Lucila Ohno-Machado,et al.  Logistic regression and artificial neural network classification models: a methodology review , 2002, J. Biomed. Informatics.

[65]  W. Knaus,et al.  The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. , 1991, Chest.

[66]  Allan Donner,et al.  Classification Efficiency of Multinomial Logistic Regression Relative to Ordinal Logistic Regression , 1989 .

[67]  M. Südkamp,et al.  Risk stratification in heart surgery: comparison of six score systems. , 2000, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[68]  S. Lemeshow,et al.  A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. , 1993, JAMA.

[69]  A. Tropsha,et al.  kappa Nearest neighbors QSAR modeling as a variational problem: theory and applications. , 2005, Journal of chemical information and modeling.

[70]  T. Higgins Quantifying risk and assessing outcome in cardiac surgery. , 1998, Journal of cardiothoracic and vascular anesthesia.

[71]  W G Henderson,et al.  Assessment of predictive models for binary outcomes: an empirical approach using operative death from cardiac surgery. , 1994, Statistics in medicine.

[72]  C D Naylor,et al.  Ready-made, recalibrated, or Remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. , 1999, Circulation.

[73]  A. H. Murphy A New Vector Partition of the Probability Score , 1973 .

[74]  Vladimir Vapnik,et al.  The Nature of Statistical Learning , 1995 .

[75]  F H Edwards,et al.  Use of a Bayesian statistical model for risk assessment in coronary artery surgery. , 1988, The Annals of thoracic surgery.

[76]  L. Joseph,et al.  Bayesian Statistics: An Introduction , 1989 .

[77]  D. Bamber The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[78]  R. Lippmann,et al.  Coronary artery bypass risk prediction using neural networks. , 1997, Annals of Thoracic Surgery.