Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

[1]  Diederick E Grobbee,et al.  Test research versus diagnostic research. , 2004, Clinical chemistry.

[2]  C. Naylor,et al.  No impact from active dissemination of the Ottawa Ankle Rules: further evidence of the need for local implementation of practice guidelines. , 1999, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[3]  Using qualitative research to inform development of a diagnostic algorithm for UTI in children. , 2013, Family practice.

[4]  Jean Sanderson,et al.  Derivation and assessment of risk prediction models using case-cohort data , 2013, BMC Medical Research Methodology.

[5]  Timothy J. Wood,et al.  Measuring Acceptability of Clinical Decision Rules: Validation of the Ottawa Acceptability of Decision Rules Instrument (OADRI) in Four Countries , 2010, Medical decision making : an international journal of the Society for Medical Decision Making.

[6]  Using the MRC Framework for Complex Interventions to Develop Clinical Decision Support: A Case Study. , 2017, Studies in health technology and informatics.

[7]  Yvonne Vergouwe,et al.  External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. , 2010, American journal of epidemiology.

[8]  B. van Calster,et al.  Visualizing Risk Prediction Models , 2015, PloS one.

[9]  M. Woodward,et al.  Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker , 2012, Heart.

[10]  Howard White,et al.  Theory-based impact evaluation: principles and practice , 2009 .

[11]  Steven D. Pearson,et al.  Physician response to a prediction rule for the triage of emergency department patients with chest pain , 1994, Journal of General Internal Medicine.

[12]  S. Cessie,et al.  Ridge Estimators in Logistic Regression , 1992 .

[13]  Ralph B D'Agostino,et al.  Presentation of multivariate data for clinical use: The Framingham Study risk score functions. , 2005, Statistics in medicine.

[14]  M. Pencina,et al.  Evaluation of Markers and Risk Prediction Models , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[15]  M. Coppieters,et al.  Interpreting research on clinical prediction rules for physiotherapy treatments. , 2011, Manual therapy.

[16]  P. Royston,et al.  Selection of important variables and determination of functional form for continuous predictors in multivariable model building , 2007, Statistics in medicine.

[17]  P. Beattie,et al.  Clinical prediction rules: what are they and what do they tell us? , 2006, The Australian journal of physiotherapy.

[18]  Matthew Thompson,et al.  Clinical prediction rules in practice: review of clinical guidelines and survey of GPs. , 2014, The British journal of general practice : the journal of the Royal College of General Practitioners.

[19]  M. Pencina,et al.  Net reclassification improvement and integrated discrimination improvement require calibrated models: relevance from a marker and model perspective , 2014, Statistics in medicine.

[20]  F. Rutten,et al.  The effects of misclassification in routine healthcare databases on the accuracy of prognostic prediction models: a case study of the CHA2DS2-VASc score in atrial fibrillation , 2017, Diagnostic and Prognostic Research.

[21]  Chava L. Ramspek,et al.  Con: Most clinical risk scores are useless. , 2017, Nephrology, dialysis, transplantation : official publication of the European Dialysis and Transplant Association - European Renal Association.

[22]  Ewout W Steyerberg,et al.  Validation in prediction research: the waste by data splitting. , 2018, Journal of clinical epidemiology.

[23]  A. Worster,et al.  Incorporation bias in studies of diagnostic tests: how to avoid being biased about bias. , 2008, CJEM.

[24]  J. Ioannidis,et al.  Reply to letter by Ferrante di Ruffano et al.: Patient outcomes in randomized comparisons of diagnostic tests are still the ultimate judge. , 2016, Journal of clinical epidemiology.

[25]  T. Lee,et al.  Evaluating decision aids , 1990, Journal of General Internal Medicine.

[26]  Pamela A Shaw,et al.  EVALUATING RISK-PREDICTION MODELS USING DATA FROM ELECTRONIC HEALTH RECORDS. , 2016, The annals of applied statistics.

[27]  R J Lilford,et al.  The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting , 2015, BMJ : British Medical Journal.

[28]  Y Vergouwe,et al.  Updating methods improved the performance of a clinical prediction model in new patients. , 2008, Journal of clinical epidemiology.

[29]  Johannes B Reitsma,et al.  The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial , 2013, BMC Cardiovascular Disorders.

[30]  K. White,et al.  Clinical epidemiology. , 1983, International journal of epidemiology.

[31]  Ewout W. Steyerberg,et al.  Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers , 2013, Statistics in medicine.

[32]  Yvonne Vergouwe,et al.  Towards better clinical prediction models: seven steps for development and an ABCD for validation. , 2014, European heart journal.

[33]  R. Perera,et al.  Why do authors derive new cardiovascular clinical prediction rules in the presence of existing rules? A mixed methods study , 2017, PloS one.

[34]  et al.,et al.  Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs) , 2011, BMC Medical Informatics Decis. Mak..

[35]  A. O’Cathain,et al.  Process evaluation of complex interventions: Medical Research Council guidance , 2015, BMJ : British Medical Journal.

[36]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[37]  P. Kranke,et al.  Quantifying prognosis with risk predictions. , 2012, European journal of anaesthesiology.

[38]  Douglas G Altman,et al.  Prognostic Models: A Methodological Framework and Review of Models for Breast Cancer , 2009, Cancer investigation.

[39]  A. Evans,et al.  Triage of patients with chest pain in the emergency department: a comparative study of physicians' decisions. , 2002, The American journal of medicine.

[40]  G. Collins,et al.  External validation of multivariable prediction models: a systematic review of methodological conduct and reporting , 2014, BMC Medical Research Methodology.

[41]  Sunil J Rao,et al.  Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis , 2003 .

[42]  G. Collins,et al.  Risk Prediction Models in Perioperative Medicine: Methodological Considerations , 2016, Current Anesthesiology Reports.

[43]  J. Graham,et al.  Missing data analysis: making it work in the real world. , 2009, Annual review of psychology.

[44]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data: Little/Statistical Analysis with Missing Data , 2002 .

[45]  D G Altman,et al.  What do we mean by validating a prognostic model? , 2000, Statistics in medicine.

[46]  G. Collins,et al.  Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting , 2011, BMC medicine.

[47]  S. Domínguez-Almendros,et al.  Logistic regression models. , 2011, Allergologia et immunopathologia.

[48]  N. Mantel Why Stepdown Procedures in Variable Selection , 1970 .

[49]  J. Ioannidis,et al.  External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. , 2015, Journal of clinical epidemiology.

[50]  Lena Osterhagen,et al.  Multiple Imputation For Nonresponse In Surveys , 2016 .

[51]  B. van Calster,et al.  Key steps and common pitfalls in developing and validating risk models , 2017, BJOG : an international journal of obstetrics and gynaecology.

[52]  I. Stiell,et al.  Use of radiography in acute ankle injuries: physicians' attitudes and practice. , 1992, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[53]  E. DeLong,et al.  Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. , 1988, Biometrics.

[54]  A Rogier T Donders,et al.  Imputation of missing values is superior to complete case analysis and the missing-indicator method in multivariable diagnostic research: a clinical example. , 2006, Journal of clinical epidemiology.

[55]  G. Guyatt,et al.  Clinical Prediction Rules , 2004 .

[56]  M. Woodward,et al.  Risk prediction models: II. External validation, model updating, and impact assessment , 2012, Heart.

[57]  Katya L. Masconi,et al.  Recalibration in Validation Studies of Diabetes Risk Prediction Models: A Systematic Review , 2015 .

[58]  Yvonne Vergouwe,et al.  Development and validation of a prediction model with missing predictor data: a practical approach. , 2010, Journal of clinical epidemiology.

[59]  Borislav D. Dimitrov,et al.  Developing an International Register of Clinical Prediction Rules for Use in Primary Care: A Descriptive Analysis , 2014, The Annals of Family Medicine.

[60]  A. Laupacis,et al.  Cost-effectiveness analysis of the Ottawa Ankle Rules. , 1995, Annals of emergency medicine.

[61]  Douglas G Altman,et al.  Dichotomizing continuous predictors in multiple regression: a bad idea , 2006, Statistics in medicine.

[62]  F. Harrell,et al.  Regression models in clinical studies: determining relationships between predictors and response. , 1988, Journal of the National Cancer Institute.

[63]  E. Steyerberg,et al.  Reporting and Methods in Clinical Prediction Research: A Systematic Review , 2012, PLoS medicine.

[64]  Yvonne Vergouwe,et al.  Bmc Medical Research Methodology Open Access Advantages of the Nested Case-control Design in Diagnostic Research , 2022 .

[65]  K. Moons,et al.  From accuracy to patient outcome and cost-effectiveness evaluations of diagnostic tests and biomarkers: an exemplary modelling study , 2013, BMC Medical Research Methodology.

[66]  Y. Vergouwe,et al.  Validation, updating and impact of clinical prediction rules: a review. , 2008, Journal of clinical epidemiology.

[67]  Ralph B D'Agostino,et al.  Misuse of DeLong test to compare AUCs for nested models , 2012, Statistics in medicine.

[68]  Tianxi Cai,et al.  The Performance of Risk Prediction Models , 2008, Biometrical journal. Biometrische Zeitschrift.

[69]  M. Cabana,et al.  Why don't physicians follow clinical practice guidelines? A framework for improvement. , 1999, JAMA.

[70]  D. Schriger,et al.  Medical decisionmaking: let's not forget the physician. , 2012, Annals of emergency medicine.

[71]  J. Griffith,et al.  Impact of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) on the speed of triage decision making for emergency department patients presenting with chest pain , 1994, Journal of General Internal Medicine.

[72]  Nathan Kuppermann,et al.  Clinical Prediction Rules for Children: A Systematic Review , 2011, Pediatrics.

[73]  S. Mullen,et al.  Qualitative analysis of clinician experience in utilising the BuRN Tool (Burns Risk assessment for Neglect or abuse Tool) in clinical practice. , 2018, Burns : journal of the International Society for Burn Injuries.

[74]  Juan Lu,et al.  Predicting Outcome after Traumatic Brain Injury: Development and International Validation of Prognostic Scores Based on Admission Characteristics , 2008, PLoS medicine.

[75]  Douglas G. Altman,et al.  No rationale for 1 variable per 10 events criterion for binary logistic regression analysis , 2016, BMC Medical Research Methodology.

[76]  Gary S Collins,et al.  A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. , 2013, Journal of clinical epidemiology.

[77]  E. Elkin,et al.  Decision Curve Analysis: A Novel Method for Evaluating Prediction Models , 2006, Medical decision making : an international journal of the Society for Medical Decision Making.

[78]  M. Katz Integrating prediction rules into clinical work flow. , 2013, JAMA internal medicine.

[79]  Dimitris Rizopoulos,et al.  Joint Models for Longitudinal and Time-to-Event Data: With Applications in R , 2012 .

[80]  Heejung Bang,et al.  How to Establish Clinical Prediction Models , 2016, Endocrinology and metabolism.

[81]  C. Kalkman,et al.  Barriers and facilitators perceived by physicians when using prediction models in practice. , 2016, Journal of clinical epidemiology.

[82]  R. Hess,et al.  Design and implementation of electronic health record integrated clinical prediction rules (iCPR): a randomized trial in diverse primary care settings , 2017, Implementation Science.

[83]  J. Kline,et al.  Clinical Decision Rules for Diagnostic Imaging in the Emergency Department: A Research Agenda. , 2015, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[84]  P. Glasziou,et al.  A Systematic Review of Studies Comparing Diagnostic Clinical Prediction Rules with Clinical Judgment , 2015, PloS one.

[85]  Yann Le Strat,et al.  Practical considerations for sensitivity analysis after multiple imputation applied to epidemiological studies with incomplete data , 2012, BMC Medical Research Methodology.

[86]  Jenny Lee,et al.  Development of a simple scoring tool in the primary care setting for prediction of recurrent falls in men and women aged 65 years and over living in the community. , 2009, Journal of clinical nursing.

[87]  K. Moons,et al.  Diagnostic and prognostic prediction models , 2013, Journal of thrombosis and haemostasis : JTH.

[88]  J. Habbema,et al.  Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. , 2001, Journal of clinical epidemiology.

[89]  D. Leys,et al.  Risk Score to Predict the Outcome of Patients with Cerebral Vein and Dural Sinus Thrombosis , 2009, Cerebrovascular Diseases.

[90]  David W. Hosmer,et al.  Best subsets logistic regression , 1989 .

[91]  A. Nierich,et al.  Prediction Models for Prolonged Intensive Care Unit Stay After Cardiac Surgery: Systematic Review and Validation Study , 2010, Circulation.

[92]  Byoung Wook Choi,et al.  How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods , 2016, Korean journal of radiology.

[93]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: what, why, and how? , 2009, BMJ : British Medical Journal.

[94]  Yvonne Vergouwe,et al.  A calibration hierarchy for risk models was defined: from utopia to empirical data. , 2016, Journal of clinical epidemiology.

[95]  Patrick Royston,et al.  Reporting methods in studies developing prognostic models in cancer: a review , 2010, BMC medicine.

[96]  C.J.H. Mann,et al.  Clinical Prediction Models: A Practical Approach to Development, Validation and Updating , 2009 .

[97]  Benjamin Brown,et al.  Understanding clinical prediction models as ‘innovations’: a mixed methods study in UK family practice , 2016, BMC Medical Informatics and Decision Making.

[98]  A. Laupacis,et al.  Emergency physicians' attitudes toward and use of clinical decision rules for radiography. , 1998, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[99]  T. Fahey,et al.  Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review , 2016, BMJ Open.

[100]  Ewout W Steyerberg,et al.  Poor performance of clinical prediction models: the harm of commonly applied methods. , 2017, Journal of clinical epidemiology.

[101]  S. Cross,et al.  Pre-endoscopy serological testing for coeliac disease: evaluation of a clinical decision tool , 2007, BMJ : British Medical Journal.

[102]  K. Covinsky,et al.  Assessing the Generalizability of Prognostic Information , 1999, Annals of Internal Medicine.

[103]  Richard D Riley,et al.  A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes , 2018, Statistical methods in medical research.

[104]  M. Kenward,et al.  An Introduction to the Bootstrap , 2007 .

[105]  Andrew Copas,et al.  Methods for sample size determination in cluster randomized trials , 2015, International journal of epidemiology.

[106]  Galit Shmueli,et al.  To Explain or To Predict? , 2010, 1101.0891.

[107]  I. Stiell,et al.  Implementation of the Ottawa ankle rules. , 1994, JAMA.

[108]  Nancy R Cook,et al.  Quantifying the added value of new biomarkers: how and how not , 2018, Diagnostic and Prognostic Research.

[109]  Qingxia Chen,et al.  Dealing with missing predictor values when applying clinical prediction models. , 2009, Clinical chemistry.

[110]  P. Glasziou,et al.  Systematic review of the effects of care provided with and without diagnostic clinical prediction rules , 2017, Diagnostic and Prognostic Research.

[111]  G W Sun,et al.  Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. , 1996, Journal of clinical epidemiology.

[112]  Gary S Collins,et al.  Sample size considerations for the external validation of a multivariable prognostic model: a resampling study , 2015, Statistics in medicine.

[113]  Rodolfo Saracci,et al.  Comprar Teaching Epidemiology A guide for teachers in epidemiology, public health and clinical medicine | Jorn Olsen | 9780199239481 | Oxford University Press , 2010 .

[114]  G. Collins,et al.  Prediction models for cardiovascular disease risk in the general population: systematic review , 2016, British Medical Journal.

[115]  B. van Calster,et al.  Calibration of Risk Prediction Models , 2015, Medical decision making : an international journal of the Society for Medical Decision Making.

[116]  Ewout W. Steyerberg,et al.  F1000Prime recommendation of Calibration of risk prediction models: impact on decision-analytic performance. , 2014 .

[117]  I. Stiell,et al.  Methodologic standards for the development of clinical decision rules in emergency medicine. , 1999, Annals of emergency medicine.

[118]  David W. Hosmer,et al.  Applied Logistic Regression , 1991 .

[119]  Katya L. Masconi,et al.  Effects of Different Missing Data Imputation Techniques on the Performance of Undiagnosed Diabetes Risk Prediction Models in a Mixed-Ancestry Population of South Africa , 2015, PloS one.

[120]  Karel G M Moons,et al.  A new framework to enhance the interpretation of external validation studies of clinical prediction models. , 2015, Journal of clinical epidemiology.

[121]  Trisha Greenhalgh,et al.  Risk models and scores for type 2 diabetes: systematic review , 2011, BMJ : British Medical Journal.

[122]  Gary S Collins,et al.  Comparing risk prediction models , 2012, BMJ : British Medical Journal.

[123]  David E. Booth,et al.  Analysis of Incomplete Multivariate Data , 2000, Technometrics.

[124]  I. Stiell,et al.  Prospective validation of a decision rule for the use of radiography in acute knee injuries. , 1996, JAMA.

[125]  A. Taylor-Vaisey,et al.  Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. , 1997, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[126]  Cody S. Olsen,et al.  Comparison of Clinician Suspicion Versus a Clinical Prediction Rule in Identifying Children at Risk for Intra-abdominal Injuries After Blunt Torso Trauma. , 2015, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[127]  W. Grobman,et al.  Methods of clinical prediction. , 2006, American journal of obstetrics and gynecology.

[128]  Charles E McCulloch,et al.  Relaxing the rule of ten events per variable in logistic and Cox regression. , 2007, American journal of epidemiology.

[129]  G. Rinkel,et al.  Decision analysis to complete diagnostic research by closing the gap between test characteristics and cost-effectiveness. , 2009, Journal of clinical epidemiology.

[130]  K Hemming,et al.  How to design efficient cluster randomised trials , 2017, British Medical Journal.

[131]  P. Royston,et al.  Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling. , 1994 .

[132]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: validating a prognostic model , 2009, BMJ : British Medical Journal.

[133]  B. Wessler,et al.  Editorial See P 332 , 2022 .

[134]  Georg Heinze,et al.  Five myths about variable selection , 2017, Transplant international : official journal of the European Society for Organ Transplantation.

[135]  T. Stijnen,et al.  Review: a gentle introduction to imputation of missing values. , 2006, Journal of clinical epidemiology.

[136]  Thomas McGinn,et al.  Putting Meaning into Meaningful Use: A Roadmap to Successful Integration of Evidence at the Point of Care , 2016, JMIR medical informatics.

[137]  Nancy R Cook,et al.  Using relative utility curves to evaluate risk prediction , 2009, Journal of the Royal Statistical Society. Series A,.

[138]  Patrick Royston,et al.  Multiple imputation using chained equations: Issues and guidance for practice , 2011, Statistics in medicine.

[139]  Douglas G Altman,et al.  Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines , 2009, BMC medical research methodology.

[140]  Michael G. Kenward,et al.  Missing data in randomised controlled trials: a practical guide , 2007 .

[141]  D E Grobbee,et al.  Assessing the applicability of scoring systems for predicting postoperative nausea and vomiting , 2005, Anaesthesia.

[142]  Davide Paolo Bernasconi,et al.  Graphical representations and summary indicators to assess the performance of risk predictors , 2018, Biometrical journal. Biometrische Zeitschrift.

[143]  Nicole A. Lazar,et al.  Statistical Analysis With Missing Data , 2003, Technometrics.

[144]  Jeremy M. Grimshaw,et al.  Changing Provider Behavior: An Overview of Systematic Reviews of Interventions , 2001, Medical care.

[145]  R. Shalvoy,et al.  The Ottawa Knee Rule: Examining Use in an Academic Emergency Department , 2012, The western journal of emergency medicine.

[146]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[147]  Eric J Topol,et al.  High-performance medicine: the convergence of human and artificial intelligence , 2019, Nature Medicine.

[148]  Gary S Collins,et al.  Prognostic models in obstetrics: available, but far from applicable. , 2016, American journal of obstetrics and gynecology.

[149]  Johann Steurer,et al.  Barriers to apply cardiovascular prediction rules in primary care: a postal survey , 2007, BMC family practice.

[150]  Ewout W Steyerberg,et al.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests , 2016, British Medical Journal.

[151]  D. Altman,et al.  CONSORT statement: extension to cluster randomised trials , 2004, BMJ : British Medical Journal.

[152]  J. Kline,et al.  Emergency medicine practitioner knowledge and use of decision rules for the evaluation of patients with suspected pulmonary embolism: variations by practice setting and training level. , 2007, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[153]  C D Naylor,et al.  Ready-made, recalibrated, or Remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. , 1999, Circulation.

[154]  James R Carpenter,et al.  Sensitivity analysis after multiple imputation under missing at random: a weighting approach , 2007, Statistical methods in medical research.

[155]  D. Farewell,et al.  Potential impact of the validated Predicting Abusive Head Trauma (PredAHT) clinical prediction tool: A clinical vignette study. , 2018, Child abuse & neglect.

[156]  P. Bossuyt,et al.  Assessing the value of diagnostic tests: a framework for designing and evaluating trials , 2012, BMJ : British Medical Journal.

[157]  M. Pencina,et al.  Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond , 2008, Statistics in medicine.

[158]  M. Ribbe,et al.  A validated risk score to estimate mortality risk in patients with dementia and pneumonia: barriers to clinical impact , 2010, International Psychogeriatrics.

[159]  W. Meurer,et al.  Logistic Regression Diagnostics: Understanding How Well a Model Predicts Outcomes , 2017, JAMA.

[160]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[161]  Y. Skaik Understanding and using sensitivity, specificity and predictive values , 2008, Indian journal of ophthalmology.

[162]  Rayid Ghani,et al.  Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness , 2018, ArXiv.

[163]  Stef van Buuren,et al.  MICE: Multivariate Imputation by Chained Equations in R , 2011 .

[164]  Diederick E Grobbee,et al.  When should we remain blind and when should our eyes remain open in diagnostic studies? , 2002, Journal of clinical epidemiology.

[165]  J. Schafer,et al.  A comparison of inclusive and restrictive strategies in modern missing data procedures. , 2001, Psychological methods.

[166]  C. Kalkman,et al.  Evaluating the impact of prediction models: lessons learned, challenges, and recommendations , 2018, Diagnostic and Prognostic Research.

[167]  L. Price,et al.  Development of a clinical prediction algorithm for knee osteoarthritis structural progression in a cohort study: value of adding measurement of subchondral bone density , 2017, Arthritis Research & Therapy.

[168]  Ian Roberts,et al.  Systematic review of prognostic models in traumatic brain injury , 2006, BMC Medical Informatics Decis. Mak..

[169]  Qingxia Chen,et al.  Missing covariate data in medical research: to impute is better than to ignore. , 2010, Journal of clinical epidemiology.

[170]  Nathan Kuppermann,et al.  Emergency physicians' knowledge and attitudes of clinical decision support in the electronic health record: a survey-based study. , 2013, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[171]  Gary S Collins,et al.  Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model , 2016, Statistics in medicine.

[172]  A. Saleh,et al.  Performance of risk assessment instruments for predicting osteoporotic fracture risk: a systematic review , 2013, Osteoporosis International.

[173]  John-Michael Sauer,et al.  Net Reclassification Index and Integrated Discrimination Index Are Not Appropriate for Testing Whether a Biomarker Improves Predictive Performance. , 2016, Toxicological sciences : an official journal of the Society of Toxicology.

[174]  L. Peelen,et al.  Prediction models: the right tool for the right problem , 2016, Current opinion in anaesthesiology.

[175]  F. Harrell,et al.  Prognostic/Clinical Prediction Models: Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors , 2005 .

[176]  H. Hutchings,et al.  Expert opinion of the risk factors for morbidity and mortality in blunt chest wall trauma: results of a national postal questionnaire survey of Emergency Departments in the United Kingdom. , 2013, Injury.

[177]  Michael J. Fine,et al.  How to derive and validate clinical prediction models for use in intensive care medicine , 2014, Intensive Care Medicine.

[178]  Thomas Agoritsas,et al.  Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. , 2011, Journal of clinical epidemiology.

[179]  Yvonne Vergouwe,et al.  Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. , 2005, Journal of clinical epidemiology.

[180]  Gary S Collins,et al.  Fracture Risk Assessment: State of the Art, Methodologically Unsound, or Poorly Reported? , 2012, Current Osteoporosis Reports.

[181]  Rodolfo Saracci,et al.  Teaching Epidemiology: A guide for teachers in epidemiology, public health and clinical medicine , 2010 .

[182]  I. Stiell,et al.  A study to develop clinical decision rules for the use of radiography in acute ankle injuries. , 1992, Annals of emergency medicine.

[183]  A. Hoes,et al.  Does a decision aid help physicians to detect chronic obstructive pulmonary disease? , 2011, The British journal of general practice : the journal of the Royal College of General Practitioners.

[184]  D. Farewell,et al.  Acceptability of the Predicting Abusive Head Trauma (PredAHT) clinical prediction tool: A qualitative study with child protection professionals. , 2018, Child abuse & neglect.

[185]  Richard D Riley,et al.  Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported. , 2018, Journal of clinical epidemiology.

[186]  J. Concato,et al.  A simulation study of the number of events per variable in logistic regression analysis. , 1996, Journal of clinical epidemiology.

[187]  A. Kemp,et al.  Exploring the acceptability of a clinical decision rule to identify paediatric burns due to child abuse or neglect , 2016, Emergency Medicine Journal.

[188]  M. Howell,et al.  Awareness and use of the Ottawa ankle and knee rules in 5 countries: can publication alone be enough to change practice? , 2001, Annals of emergency medicine.

[189]  Mithat Gönen,et al.  A new concordance measure for risk prediction models in external validation settings , 2016, Statistics in medicine.

[190]  I. Stiell,et al.  Will a new clinical decision rule be widely used? The case of the Canadian C-spine rule. , 2006, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[191]  D E Grobbee,et al.  External validation is necessary in prediction research: a clinical example. , 2003, Journal of clinical epidemiology.

[192]  Joseph L Schafer,et al.  On the performance of random‐coefficient pattern‐mixture models for non‐ignorable drop‐out , 2003, Statistics in medicine.

[193]  R. Perera,et al.  Predictors for independent external validation of cardiovascular risk clinical prediction rules: Cox proportional hazards regression analyses , 2018, Diagnostic and Prognostic Research.

[194]  D. Rivett,et al.  Australian physiotherapists' priorities for the development of clinical prediction rules for low back pain: a qualitative study. , 2015, Physiotherapy.

[195]  M. Levine,et al.  Systematic review of clinical prediction tools and prognostic factors in aneurysmal subarachnoid hemorrhage , 2015, Surgical neurology international.

[196]  Ewout W Steyerberg,et al.  Validation and updating of predictive logistic regression models: a study on sample size and shrinkage , 2004, Statistics in medicine.

[197]  Gareth Ambler,et al.  How to develop a more accurate risk prediction model when there are few events , 2015, BMJ : British Medical Journal.

[198]  Gary S Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration , 2015, Annals of Internal Medicine.

[199]  A. Laupacis,et al.  Clinical prediction rules. A review and suggested modifications of methodological standards. , 1997, JAMA.

[200]  Jing Fan,et al.  The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets , 2015, Statistics in biosciences.

[201]  P. Royston,et al.  Prognosis and prognostic research: application and impact of prognostic models in clinical practice , 2009, BMJ : British Medical Journal.

[202]  K. Tremper,et al.  Development and Validation of an Acute Kidney Injury Risk Index for Patients Undergoing General Surgery: Results from a National Data Set , 2009, Anesthesiology.

[203]  E W Steyerberg,et al.  Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. , 1999, Journal of clinical epidemiology.

[204]  Stavros Petrou,et al.  Economic evaluation using decision analytical modelling: design, conduct, analysis, and reporting , 2011, BMJ : British Medical Journal.

[205]  Donald M Yealy,et al.  Methodologic standards for interpreting clinical decision rules in emergency medicine: 2014 update. , 2014, Annals of emergency medicine.

[206]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[207]  T. Cole,et al.  SCALING AND ROUNDING REGRESSION-COEFFICIENTS TO INTEGERS , 1993 .

[208]  Maarten van Smeden,et al.  Sample size for binary logistic prediction models: Beyond events per variable criteria , 2018, Statistical methods in medical research.

[209]  R. Roberts,et al.  Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. , 2002, JAMA.

[210]  C. Maher,et al.  A Guide to Interpretation of Studies Investigating Subgroups of Responders to Physical Therapy Interventions , 2009, Physical Therapy.

[211]  Susan Michie,et al.  Changing clinical behaviour by making guidelines specific , 2004, BMJ : British Medical Journal.

[212]  E. Steyerberg,et al.  Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research , 2013, PLoS medicine.

[213]  Tom Fahey,et al.  Clinical prediction rules in primary care: what can be done to maximise their implementation? , 2010 .

[214]  H. Sox,et al.  Clinical prediction rules. Applications and methodological standards. , 1985, The New England journal of medicine.

[215]  A. Evans,et al.  Translating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions , 2006, Annals of Internal Medicine.

[216]  J. Habbema,et al.  Prognostic Modeling with Logistic Regression Analysis , 2001, Medical decision making : an international journal of the Society for Medical Decision Making.

[217]  Douglas G. Altman,et al.  Adequate sample size for developing prediction models is not simply related to events per variable , 2016, Journal of clinical epidemiology.

[218]  G. Collins,et al.  Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist , 2014, PLoS medicine.

[219]  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2011, BMJ : British Medical Journal.

[220]  S. Sanders,et al.  Clinical prediction rules for assisting diagnosis , 2015 .

[221]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[222]  C. Kalkman,et al.  Impact of adding therapeutic recommendations to risk assessments from a prediction model for postoperative nausea and vomiting. , 2015, British journal of anaesthesia.

[223]  M. Kenward,et al.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls , 2009, BMJ : British Medical Journal.

[224]  Theo Stijnen,et al.  Using the outcome for imputation of missing predictor values was preferred. , 2006, Journal of clinical epidemiology.

[225]  Douglas G Altman,et al.  Comparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study , 2010, BMC medical research methodology.

[226]  Shahrokh F. Shariat,et al.  Inventory of prostate cancer predictive tools , 2008, Current opinion in urology.

[227]  M. Leeflang,et al.  Search Filters for Finding Prognostic and Diagnostic Prediction Studies in Medline to Enhance Systematic Reviews , 2012, PloS one.

[228]  I. Stiell,et al.  Clinical decision rules "in the real world": how a widely disseminated rule is used in everyday practice. , 2005, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[229]  G H Guyatt,et al.  Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. , 2000, JAMA.

[230]  Suzanne Bakken,et al.  Informing the design of clinical decision support services for evaluation of children with minor blunt head trauma in the emergency department: A sociotechnical analysis , 2013, J. Biomed. Informatics.

[231]  Ewout W Steyerberg,et al.  Logistic regression modeling and the number of events per variable: selection bias dominates. , 2011, Journal of clinical epidemiology.

[232]  Yvonne Vergouwe,et al.  Prognosis and prognostic research: Developing a prognostic model , 2009, BMJ : British Medical Journal.

[233]  Michael J Pencina,et al.  Evaluating Discrimination of Risk Prediction Models: The C Statistic. , 2015, JAMA.

[234]  Patrick Royston,et al.  Multivariable Model-Building: A Pragmatic Approach to Regression Analysis based on Fractional Polynomials for Modelling Continuous Variables , 2008 .

[235]  R Z Omar,et al.  An evaluation of penalised survival methods for developing prognostic models with rare events , 2012, Statistics in medicine.

[236]  Carol Bennett,et al.  Implementation of clinical decision rules in the emergency department. , 2007, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[237]  Kjetil Søreide,et al.  Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research , 2008, Journal of Clinical Pathology.

[238]  U. Narayanan,et al.  Pediatric emergency physician opinions on ankle radiograph clinical decision rules. , 2010, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[239]  John P A Ioannidis,et al.  Comparisons of established risk prediction models for cardiovascular disease: systematic review , 2012, BMJ : British Medical Journal.

[240]  Douglas G. Altman,et al.  Improving the Transparency of Prognosis Research: The Role of Reporting, Data Sharing, Registration, and Protocols , 2014, PLoS medicine.

[241]  Harvey Goldstein,et al.  Multilevel models with multivariate mixed response types , 2009 .

[242]  Thomas A Gerds,et al.  A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index , 2014, Statistics in medicine.

[243]  E W Steyerberg,et al.  Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective , 2019, Statistics in medicine.

[244]  M. Ebell,et al.  A novel approach to the determination of clinical decision thresholds , 2015, Evidence-Based Medicine.

[245]  Tra My Pham,et al.  Missing data and multiple imputation in clinical epidemiological research , 2017, Clinical epidemiology.

[246]  Douglas G Altman,et al.  The time has come to register diagnostic and prognostic research. , 2014, Clinical chemistry.

[247]  M. Mackey,et al.  Health practitioners’ perceptions of adopting clinical prediction rules in the management of musculoskeletal pain: a qualitative study in Australia , 2017, BMJ Open.

[248]  H. Burchardi,et al.  Outcome prediction in critical care: physicians' prognoses vs. scoring systems , 2004, European journal of anaesthesiology.

[249]  R Henderson,et al.  Joint modelling of longitudinal measurements and event time data. , 2000, Biostatistics.

[250]  M. Pencina,et al.  Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. , 2014, Annals of Internal Medicine.

[251]  Ian R White,et al.  Analyses of Sensitivity to the Missing-at-Random Assumption Using Multiple Imputation With Delta Adjustment: Application to a Tuberculosis/HIV Prevalence Survey With Incomplete HIV-Status Data , 2017, American journal of epidemiology.

[252]  R. Marshall The use of classification and regression trees in clinical epidemiology. , 2001, Journal of clinical epidemiology.

[253]  D. Rivett,et al.  Physiotherapists' knowledge, attitudes and practices regarding clinical prediction rules for low back pain. , 2014, Manual therapy.

[254]  D. Rubin,et al.  Statistical Analysis with Missing Data. , 1989 .

[255]  P. Dayan,et al.  Comparison of Prediction Rules and Clinician Suspicion for Identifying Children With Clinically Important Brain Injuries After Blunt Head Trauma. , 2016, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[256]  Yvonne Vergouwe,et al.  Adaptation of Clinical Prediction Models for Application in Local Settings , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[257]  Karel Moons,et al.  PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration , 2019, Annals of Internal Medicine.