Evaluating the Impact of Uncertainty on Risk Prediction: Towards More Robust Prediction Models

Risk prediction models are crucial for assessing the pretest probability of cancer and are applied to stratify patient management strategies. These models are frequently based on multivariate regression analysis, requiring that all risk factors be specified, and do not convey the confidence in their predictions. We present a framework for uncertainty analysis that accounts for variability in input values. Uncertain or missing values are replaced with a range of plausible values. These ranges are used to compute individualized risk confidence intervals. We demonstrate our approach using the Gail model to evaluate the impact of uncertainty on management decisions. Up to 13% of cases (uncertain) had a risk interval that falls within the decision threshold (e.g., 1.67% 5-year absolute risk). A small number of cases changed from low- to high-risk when missing values were present. Our analysis underscores the need for better communication of input assumptions that influence the resulting predictions.

[1]  Laura Esserman,et al.  Rethinking screening for breast cancer and prostate cancer. , 2009, JAMA.

[2]  Mitchell H Gail,et al.  Personalized estimates of breast cancer risk in clinical practice and public health , 2011, Statistics in medicine.

[3]  M. Kenward,et al.  Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls , 2009, BMJ : British Medical Journal.

[4]  C K Redmond,et al.  Tamoxifen for prevention of breast cancer: report of the National Surgical Adjuvant Breast and Bowel Project P-1 Study. , 1999, Journal of the National Cancer Institute.

[5]  Rashid Al-Abri,et al.  Patient satisfaction survey as a tool towards quality improvement. , 2014, Oman medical journal.

[6]  Stephen W Duffy,et al.  A breast cancer prediction model incorporating familial and personal risk factors , 2004, Hereditary Cancer in Clinical Practice.

[7]  Nadine B. Sarter,et al.  Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information , 2006, Hum. Factors.

[8]  L. Walter,et al.  Cancer screening in elderly patients: a framework for individualized decision making. , 2001, JAMA.

[9]  W Jorritsma,et al.  Improving the radiologist-CAD interaction: designing for appropriate trust. , 2015, Clinical radiology.

[10]  Jason Roy,et al.  Missing laboratory results data in electronic health databases: implications for monitoring diabetes risk. , 2017, Journal of comparative effectiveness research.

[11]  P. Slovic,et al.  Numeracy skill and the communication, comprehension, and use of risk-benefit information. , 2007, Health affairs.

[12]  Andrew N Freedman,et al.  Communication of Uncertainty Regarding Individualized Cancer Risk Estimates , 2011, Medical decision making : an international journal of the Society for Medical Decision Making.

[13]  M. Gail,et al.  Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. , 1989, Journal of the National Cancer Institute.

[14]  Gordon B Mills,et al.  Pretest prediction of BRCA1 or BRCA2 mutation by risk counselors and the computer model BRCAPRO. , 2002, Journal of the National Cancer Institute.

[15]  R. Coombes,et al.  Ahead of the game , 2008, BMJ : British Medical Journal.

[16]  Andrew N Freedman,et al.  Cancer risk prediction models: a workshop on development, evaluation, and application. , 2005, Journal of the National Cancer Institute.

[17]  J Benichou,et al.  Validation studies for models projecting the risk of invasive and total breast cancer incidence. , 1999, Journal of the National Cancer Institute.