Calculating the sample size required for developing a clinical prediction model

Clinical prediction models aim to predict outcomes in individuals, to inform diagnosis or prognosis in healthcare. Hundreds of prediction models are published in the medical literature each year, yet many are developed using a dataset that is too small for the total number of participants or outcome events. This leads to inaccurate predictions and consequently incorrect healthcare decisions for some individuals. In this article, the authors provide guidance on how to calculate the sample size required to develop a clinical prediction model.

[1]  Karel G M Moons,et al.  A closed testing procedure to select an appropriate method for updating prediction models , 2017, Statistics in medicine.

[2]  D. McFadden.  Conditional logit analysis of qualitative choice behavior , 1972 .

[3]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[4]  Joseph R. Rausch,et al.  Sample size planning for statistical power and accuracy in parameter estimation. , 2008, Annual review of psychology.

[5]  Frank E. Harrell,et al.  Prediction models need appropriate internal, internal-external, and external validation. , 2016, Journal of clinical epidemiology.

[6]  Yvonne Vergouwe,et al.  A calibration hierarchy for risk models was defined: from utopia to empirical data. , 2016, Journal of clinical epidemiology.

[7]  G. Collins,et al.  PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies , 2019, Annals of Internal Medicine.

[8]  J. Concato,et al.  A simulation study of the number of events per variable in logistic regression analysis. , 1996, Journal of clinical epidemiology.

[9]  Ken Kelley,et al.  Sample size for multiple regression: obtaining regression coefficients that are accurate, not simply significant. , 2003, Psychological methods.

[10]  J. Copas,et al.  Using regression models for prediction: shrinkage and regression to the mean , 1997, Statistical methods in medical research.

[11]  Karel G M Moons,et al.  Meta‐analysis and aggregation of multiple published prediction models , 2014, Statistics in medicine.

[12]  P Peduzzi,et al.  Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy. , 1995, Journal of clinical epidemiology.

[13]  Karel Moons,et al.  PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration , 2019, Annals of Internal Medicine.

[14]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[15]  R. Blamey,et al.  A prognostic index in primary breast cancer. , 1982, British Journal of Cancer.

[16]  Gowri Raman,et al.  Tufts PACE Clinical Predictive Model Registry: update 1990 through 2015 , 2017, Diagnostic and Prognostic Research.

[17]  Yvonne Vergouwe,et al.  Incorporating published univariable associations in diagnostic and prognostic modeling , 2012, BMC Medical Research Methodology.

[18]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[19]  Gareth Ambler,et al.  How to develop a more accurate risk prediction model when there are few events , 2015, BMJ : British Medical Journal.

[20]  Gary S Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration , 2015, Annals of Internal Medicine.

[21]  L. Hooft,et al.  A guide to systematic review and meta-analysis of prediction model performance , 2017, British Medical Journal.

[22]  Johannes B. Reitsma,et al.  Individual Participant Data (IPD) Meta-analyses of Diagnostic and Prognostic Modeling Studies: Guidance on Their Use , 2015, PLoS medicine.

[23]  J. Copas.  Regression, Prediction and Shrinkage , 1983 .

[24]  P. Austin,et al.  Events per variable (EPV) and the relative performance of different strategies for estimating the out-of-sample validity of logistic regression models , 2014, Statistical methods in medical research.

[25]  Janis Bormanis,et al.  Value of assessment of pretest probability of deep-vein thrombosis in clinical management , 1997, The Lancet.

[26]  E. Steyerberg.  Clinical Prediction Models , 2008, Statistics for Biology and Health.

[27]  Georg Heinze,et al.  Variable selection – A review and recommendations for the practicing statistician , 2018, Biometrical journal. Biometrische Zeitschrift.

[28]  M Gent,et al.  Derivation of a Simple Clinical Model to Categorize Patients Probability of Pulmonary Embolism: Increasing the Models Utility with the SimpliRED D-dimer , 2000, Thrombosis and Haemostasis.

[29]  Frank E. Harrell,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2001 .

[30]  Gary S Collins,et al.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study , 2015, Statistics in medicine.

[31]  Richard D Riley,et al.  External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges , 2016, BMJ.

[32]  John O'Quigley,et al.  Explained randomness in proportional hazards models , 2005, Statistics in medicine.

[33]  Maarten van Smeden,et al.  Sample size for binary logistic prediction models: Beyond events per variable criteria , 2018, Statistical methods in medical research.

[34]  Anne-Laure Boulesteix,et al.  Stability Investigations of Multivariable Regression Models Derived from Low- and High-Dimensional Data , 2011, Journal of biopharmaceutical statistics.

[35]  Gary H. McClelland,et al.  Increasing statistical power without increasing sample size. , 2000 .

[36]  Daniel B. Mark,et al.  TUTORIAL IN BIOSTATISTICS MULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS , 1996 .

[37]  Richard D Riley,et al.  University of Birmingham A systematic review of prognostic models for recurrent venous thromboembolism (VTE) post treatment of first unprovoked VTE , 2016 .

[38]  Douglas G. Altman,et al.  No rationale for 1 variable per 10 events criterion for binary logistic regression analysis , 2016, BMC Medical Research Methodology.

[39]  J. Habbema,et al.  Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. , 2001, Journal of clinical epidemiology.

[40]  Maarten van Smeden,et al.  Sample size considerations and predictive performance of multinomial logistic prediction models , 2019, Statistics in medicine.

[41]  Patrick Royston,et al.  A new measure of prognostic separation in survival data , 2004, Statistics in medicine.

[42]  I. Ellis,et al.  The Nottingham prognostic index in primary breast cancer , 2005, Breast Cancer Research and Treatment.

[43]  Douglas G. Altman,et al.  Adequate sample size for developing prediction models is not simply related to events per variable , 2016, Journal of clinical epidemiology.

[44]  Richard D Riley,et al.  Minimum sample size for developing a multivariable prediction model: PART II ‐ binary and time‐to‐event outcomes , 2018, Statistics in medicine.

[45]  Ewout W Steyerberg,et al.  Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints , 2014, BMC Medical Research Methodology.

[46]  P. Royston,et al.  Stability of multivariable fractional polynomial models with selection of variables and transformations: a bootstrap investigation , 2003, Statistics in medicine.

[47]  Richard D Riley,et al.  Minimum sample size for developing a multivariable prediction model: Part I – Continuous outcomes , 2018, Statistics in medicine.

[48]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[49]  Thomas Agoritsas,et al.  Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. , 2011, Journal of clinical epidemiology.

[50]  Yvonne Vergouwe,et al.  Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. , 2005, Journal of clinical epidemiology.

[51]  J. Hippisley-Cox,et al.  Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study , 2017, British Medical Journal.

[52]  Ewout W Steyerberg,et al.  Risk prediction with machine learning and regression methods , 2014, Biometrical journal. Biometrische Zeitschrift.

[53]  J. C. van Houwelingen,et al.  Shrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy , 2001 .

[54]  D. Bloch,et al.  A simple method of sample size calculation for linear and logistic regression. , 1998, Statistics in medicine.

[55]  Charles E McCulloch,et al.  Relaxing the rule of ten events per variable in logistic and Cox regression. , 2007, American journal of epidemiology.

[56]  Harvey J Cohen,et al.  An Overview of Variance Inflation Factors for Sample-Size Calculation , 2003, Evaluation & the health professions.

[57]  Ken Kelley,et al.  Sample size planning for the coefficient of variation from the accuracy in parameter estimation approach , 2007, Behavior research methods.

[58]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[59]  P W Lavori,et al.  Sample-size calculations for the Cox proportional hazards regression model with nonbinary covariates. , 2000, Controlled clinical trials.

[60]  Sander Greenland,et al.  Sparse data bias: a problem hiding in plain sight , 2016, British Medical Journal.

[61]  S. le Cessie,et al.  Predictive value of statistical models. , 1990, Statistics in medicine.

[62]  M Schumacher,et al.  Sample size considerations for the evaluation of prognostic factors in survival analysis. , 2000, Statistics in medicine.

[63]  Richard D Riley,et al.  Development and validation of a prediction model for fat mass in children and adolescents: meta-analysis using individual participant data , 2019, BMJ.

[64]  David R. Cox.  The analysis of binary data , 1970 .

[65]  Richard D Riley,et al.  Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluation. , 2016, Health technology assessment.

[66]  Richard D Riley,et al.  Prognosis research ideally should measure time-varying predictors at their intended moment of use , 2017, Diagnostic and Prognostic Research.

[67]  S Van Huffel,et al.  A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data. , 2015, Journal of clinical epidemiology.

[68]  Karel G M Moons,et al.  Aggregating published prediction models with individual participant data: a comparison of different approaches , 2012, Statistics in medicine.

[69]  Michael A Black,et al.  Clinical risk prediction for pre-eclampsia in nulliparous women: development of model in international prospective cohort , 2011, BMJ : British Medical Journal.

[70]  K. Anderson,et al.  Cardiovascular disease risk profiles. , 1991, American heart journal.

[71]  Patrick Royston,et al.  Explained Variation for Survival Models , 2006 .

[72]  N. Nagelkerke,et al.  A note on a general definition of the coefficient of determination , 1991 .

[73]  M Schumacher,et al.  A bootstrap resampling procedure for model building: application to the Cox regression model. , 1992, Statistics in medicine.

[74]  Ewout W Steyerberg,et al.  The number of subjects per variable required in linear regression analyses. , 2015, Journal of clinical epidemiology.

[75]  James E. Helmreich.  Regression Modeling Strategies with Applications to Linear Models, Logistic and Ordinal Regression and Survival Analysis (2nd Edition) , 2016 .

[76]  J. Concato,et al.  Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. , 1995, Journal of clinical epidemiology.

[77]  Patrick Royston,et al.  Discrimination-based sample size calculations for multivariable prognostic models for time-to-event data , 2015, BMC Medical Research Methodology.

[78]  Richard D Riley,et al.  A guide to systematic review and meta-analysis of prognostic factor studies , 2019, British Medical Journal.

[79]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .