Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.

CONTEXT Urologists regularly develop clinical risk prediction models to support clinical decisions. In contrast to traditional performance measures, decision curve analysis (DCA) can assess the utility of models for decision making. DCA plots net benefit (NB) at a range of clinically reasonable risk thresholds. OBJECTIVE To provide recommendations on interpreting and reporting DCA when evaluating prediction models. EVIDENCE ACQUISITION We informally reviewed the urological literature to determine investigators' understanding of DCA. To illustrate, we use data from 3616 patients to develop risk models for high-grade prostate cancer (n=313, 9%) to decide who should undergo a biopsy. The baseline model includes prostate-specific antigen and digital rectal examination; the extended model adds two predictors based on transrectal ultrasound (TRUS). EVIDENCE SYNTHESIS We explain risk thresholds, NB, default strategies (treat all, treat no one), and test tradeoff. To use DCA, first determine whether a model is superior to all other strategies across the range of reasonable risk thresholds. If so, that model appears to improve decisions irrespective of threshold. Second, consider if there are important extra costs to using the model. If so, obtain the test tradeoff to check whether the increase in NB versus the best other strategy is worth the additional cost. In our case study, addition of TRUS improved NB by 0.0114, equivalent to 1.1 more detected high-grade prostate cancers per 100 patients. Hence, adding TRUS would be worthwhile if we accept subjecting 88 patients to TRUS to find one additional high-grade prostate cancer or, alternatively, subjecting 10 patients to TRUS to avoid one unnecessary biopsy. CONCLUSIONS The proposed guidelines can help researchers understand DCA and improve application and reporting. PATIENT SUMMARY Decision curve analysis can identify risk models that can help us make better clinical decisions. We illustrate appropriate reporting and interpretation of decision curve analysis.

[1]  Mithat Gonen,et al.  Nomograms in oncology: more than meets the eye. , 2015, The Lancet. Oncology.

[2]  Geert Trooskens,et al.  Detection of High-grade Prostate Cancer Using a Urinary Molecular Biomarker-Based Risk Score. , 2016, European urology.

[3]  Ewout W Steyerberg,et al.  Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers , 2011, Statistics in medicine.

[4]  P. Scardino,et al.  Critical review of prostate cancer predictive tools. , 2009, Future oncology.

[5]  B. van Calster,et al.  Calibration of Risk Prediction Models , 2015, Medical decision making : an international journal of the Society for Medical Decision Making.

[6]  G. Collins,et al.  Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement , 2015, BMJ : British Medical Journal.

[7]  S. Goodman,et al.  Beyond the Usual Prediction Accuracy Metrics: Reporting Results for Clinical Decision Making , 2012, Annals of Internal Medicine.

[8]  Valentin Rousson,et al.  Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies , 2011, BMC Medical Informatics Decis. Mak..

[9]  Joshua T. Cohen,et al.  Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models , 2017, Medical decision making : an international journal of the Society for Medical Decision Making.

[10]  E. Elkin,et al.  Decision Curve Analysis: A Novel Method for Evaluating Prediction Models , 2006, Medical decision making : an international journal of the Society for Medical Decision Making.

[11]  A. Vickers,et al.  Evaluation of Prediction Models for Decision-Making: Beyond Calibration and Discrimination , 2013, PLoS medicine.

[12]  N. Obuchowski,et al.  Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures , 2010, Epidemiology.

[13]  Chris Hyde,et al.  Decision Making in Health and Medicine. Integrating Evidence and Values , 2005, ACP Journal Club.

[14]  T. Tammela,et al.  Screening and prostate cancer mortality: results of the European Randomised Study of Screening for Prostate Cancer (ERSPC) at 13 years of follow-up , 2014, The Lancet.

[15]  J. Kassirer,et al.  Therapeutic decision making: a cost-benefit analysis. , 1975, The New England journal of medicine.

[16]  J Hilden.  Prevalence-free utility-respecting summary indices of diagnostic power do not exist. , 2000, Statistics in medicine.

[17]  M. Roethke,et al.  Combined Clinical Parameters and Multiparametric Magnetic Resonance Imaging for Advanced Risk Modeling of Prostate Cancer-Patient-tailored Risk Stratification Can Reduce Unnecessary Biopsies. , 2017, European urology.

[18]  Kathleen F. Kerr,et al.  Assessing the Clinical Impact of Risk Prediction Models With Decision Curves: Guidance for Correct Interpretation and Appropriate Use. , 2016, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[19]  M. Pencina,et al.  How to interpret a small increase in AUC with an additional risk prediction marker: decision analysis comes through , 2014, Statistics in medicine.

[20]  Yvonne Vergouwe,et al.  A calibration hierarchy for risk models was defined: from utopia to empirical data. , 2016, Journal of clinical epidemiology.

[21]  John B. Wong,et al.  Decision making in health and medicine: Integrating evidence and values, second edition , 2014 .

[22]  Gary S Collins,et al.  Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration , 2015, Annals of Internal Medicine.

[23]  Nancy R Cook,et al.  Using relative utility curves to evaluate risk prediction , 2009, Journal of the Royal Statistical Society. Series A,.

[24]  Holly Janes,et al.  Net risk reclassification p values: valid or misleading? , 2014, Journal of the National Cancer Institute.

[25]  A. Vickers.  Prediction models: revolutionary in principle, but do they do more good than harm? , 2011, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[26]  E. Steyerberg,et al.  Prediction of prostate cancer risk: the role of prostate volume and digital rectal examination in the ERSPC risk calculators. , 2012, European urology.

[27]  Ewout W. Steyerberg,et al.  F1000Prime recommendation of Calibration of risk prediction models: impact on decision-analytic performance. , 2014 .

[28]  H. G. van der Poel,et al.  EAU-ESTRO-SIOG Guidelines on Prostate Cancer. Part 1: Screening, Diagnosis, and Local Treatment with Curative Intent. , 2017, European urology.

[29]  N. Perkins,et al.  The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. , 2006, American journal of epidemiology.

[30]  Ewout W Steyerberg,et al.  Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests , 2016, British Medical Journal.

[31]  Benjamin R Saville,et al.  Decision curve analysis. , 2015, JAMA.

[32]  Andrew J Vickers.  Incorporating Clinical Considerations into Statistical Analyses of Markers: A Quiet Revolution in How We Think About Data. , 2016, Clinical chemistry.

[33]  Ewout W Steyerberg,et al.  A risk-based strategy improves prostate-specific antigen-driven detection of prostate cancer. , 2010, European urology.

[34]  E. Steyerberg.  Clinical Prediction Models , 2008, Statistics for Biology and Health.

[35]  M. Pencina,et al.  Evaluation of Markers and Risk Prediction Models , 2013, Medical decision making : an international journal of the Society for Medical Decision Making.

[36]  F. Harrell,et al.  Criteria for Evaluation of Novel Markers of Cardiovascular Risk: A Scientific Statement From the American Heart Association , 2009, Circulation.

[37]  Andrew J Vickers,et al.  Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). , 2010, Urology.

[38]  Prospective evaluation on the effect of interobserver variability of digital rectal examination on the performance of the Rotterdam Prostate Cancer Risk Calculator , 2017, International journal of urology : official journal of the Japanese Urological Association.

[39]  B. van Calster,et al.  The International Journal of Biostatistics Evaluating a New Marker for Risk Prediction Using the Test Tradeoff : An Update , 2012 .

[40]  B. Kramer,et al.  Evaluating Prognostic Markers Using Relative Utility Curves and Test Tradeoffs. , 2015, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.