Statistical process control for validating a classification tree model for predicting mortality - A novel approach towards temporal validation

Prediction models are postulated as useful tools to support tasks such as clinical decision making and benchmarking. In particular, classification tree models have enjoyed much interest in the Biomedical Informatics literature. However, their prospective predictive performance over the course of time has not been investigated. In this paper we suggest and apply statistical process control methods to monitor over more than 5 years the prospective predictive performance of TM80+, one of the few classification-tree models published in the clinical literature. TM80+ is a model for predicting mortality among very elderly patients in the intensive care based on a multi-center dataset. We also inspect the predictive performance at the tree's leaves. This study provides important insights into patterns of (in)stability of the tree's performance and its "shelf life". The study underlies the importance of continuous validation of prognostic models over time using statistical tools and the timely recalibration of tree models.

[1]  A Abu-Hanna,et al.  Prognostic methods in medicine. , 1999, Artificial intelligence in medicine.

[2]  Ameen Abu-Hanna,et al.  Characterizing Predictive Models of Mortality for Older Adults and Their Validation for Use in Clinical Practice , 2011, Journal of the American Geriatrics Society.

[3]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: validating a prognostic model , 2009, BMJ : British Medical Journal.

[4]  David J. Hand,et al.  Construction and Assessment of Classification Rules , 1997 .

[5]  A. Abu-Hanna,et al.  Identification of high-risk subgroups in very elderly intensive care unit patients , 2007, Critical care.

[6]  John Chuo.  Dynamic risk adjustment of prediction models using statistical process control methods , 2004 .

[7]  C.J.H. Mann,et al.  Clinical Prediction Models: A Practical Approach to Development, Validation and Updating , 2009 .

[8]  Raymond G. Carey,et al.  How Do You Know That Your Care Is Improving? Part I: Basic Concepts in Statistical Thinking , 2002 .

[9]  A. Abu-Hanna,et al.  Prognostic Models in Medicine , 2001, Methods of Information in Medicine.

[10]  Petra M. Kuhnert,et al.  Application and evaluation of classification trees for screening unwanted plants , 2006 .

[11]  Niall M. Adams,et al.  The impact of changing populations on classifier performance , 1999, KDD '99.

[12]  R. D'Agostino,et al.  External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks. , 2003, Journal of clinical epidemiology.

[13]  E W Steyerberg,et al.  Identifying subgroups among poor prognosis patients with nonseminomatous germ cell cancer by tree modelling: a validation study. , 2004, Annals of oncology : official journal of the European Society for Medical Oncology.

[14]  M. Silver,et al.  Evaluation of 6 prognostic models used to calculate mortality rates in elderly heart failure patients with a fatal heart failure admission. , 2010, Congestive heart failure.

[15]  D. Cook,et al.  Methods to Assess Performance of Models Estimating Risk of Death in Intensive Care Patients: A Review , 2006, Anaesthesia and intensive care.

[16]  Roberta F. White,et al.  Repeated split sample validation to assess logistic regression and recursive partitioning: an application to the prediction of cognitive impairment , 2005, Statistics in medicine.

[17]  Pawan Sikka,et al.  Outcome of Older Patients with Severe Pneumonia Predicted by Recursive Partitioning , 2001, Journal of the American Geriatrics Society.

[18]  Marcus B. Perry,et al.  The Exponentially Weighted Moving Average , 2010 .

[19]  Iie Arnold L. Sweet Senior Member.  Control Charts Using Coupled Exponentially Weighted Moving Averages , 1986 .

[20]  Stefan Conrad,et al.  Prospective validation of an algorithm with systematic sextant biopsy to predict pelvic lymph node metastasis in patients with clinically localized prostatic carcinoma. , 2002, The Journal of urology.

[21]  P. Robinson,et al.  Average Run Lengths of Geometric Moving Average Charts by Numerical Methods , 1978 .

[22]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: what, why, and how? , 2009, BMJ : British Medical Journal.

[23]  D. Hosmer,et al.  A comparison of goodness-of-fit tests for the logistic regression model. , 1997, Statistics in medicine.

[24]  Arthur B. Yeh,et al.  Exponentially Weighted Moving Average (EWMA) Control Charts for Monitoring an Analytical Process , 2008 .

[25]  Benjamin M. Adams,et al.  Advanced Topics in Statistical Process Control : The Power of Shewhart's Charts , 1995 .

[26]  Peter C Austin,et al.  A comparison of regression trees, logistic regression, generalized additive models, and multivariate adaptive regression splines for predicting AMI mortality , 2007, Statistics in medicine.

[27]  Karen Kafadar.  Statistical Process Control: The Deming Paradigm and Beyond , 2003, Technometrics.

[28]  D. Koller,et al.  Integration of Early Physiological Responses Predicts Later Illness Severity in Preterm Infants , 2010, Science Translational Medicine.

[29]  James Jaccard,et al.  A decision theoretic and prototype conceptualization of possible selves: implications for the prediction of risk behavior. , 2006, Journal of personality.

[30]  N F de Keizer,et al.  [Intensive care medicine in the Netherlands, 1997-2001. I. Patient population and treatment outcome]. , 2003, Nederlands tijdschrift voor geneeskunde.

[31]  Rose M. Ray,et al.  Assessment of the Beryllium Lymphocyte Proliferation Test Using Statistical Process Control , 2006, Inhalation toxicology.

[32]  M A Mohammed,et al.  Plotting basic control charts: tutorial notes for healthcare practitioners , 2008, Quality & Safety in Health Care.

[33]  Alexey Tsymbal,et al.  The problem of concept drift: definitions and related work , 2004 .

[34]  J. Wyatt,et al.  Commentary: Prognostic models: clinically useful or quickly forgotten? , 1995 .

[35]  R. Marshall.  The use of classification and regression trees in clinical epidemiology. , 2001, Journal of clinical epidemiology.

[36]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[37]  van Zanten Ar,et al.  [Intensive care medicine in the Netherlands, 1997-2001]. , 2003 .