Measuring the accuracy of prognostic judgments in oncology.

The purpose of this study was to describe the accuracy of prognostic judgments in the day-to-day practice of cancer medicine. Oncologists' initial estimates of the probability of cure of 98 cancer patients, and of the duration of survival of 39 incurable cancer patients, were compared with the observed outcome at five years. Estimates of probability of cure were well calibrated and also relatively accurate (Mean Brier Accuracy Score [AS] = 0.11 +/- 0.02 [SE]). Oncologists' ability to discriminate between curable and incurable patients compared favorably with the discrimination of prognostic judgments in acute care medicine (area under the receiver operating characteristic curve, AROC = 0.91 +/- 0.09). Estimates of the duration of survival of incurable patients were well calibrated, but individual predictions were imprecise. Discrimination between patients who would survive for three months, and those who would not, was only fair (AROC = 0.75 +/- 0.04), and discrimination between patients who would survive for one year and those who would not was very poor (AROC = 0.57 +/- 0.01). It was concluded that oncologists' estimates of probability of cure are surprisingly accurate, but estimates of the duration of survival of incurable patients are much less accurate, and subject to a "horizon effect" similar to that recognized in weather forecasting.

[1]  J G Dolan,et al.  An Eualuation of Clinicians' Subjective Prior Probability Estimates , 1986, Medical decision making : an international journal of the Society for Medical Decision Making.

[2]  M. Mccarthy,et al.  PROGNOSTIC UNCERTAINTY IN TERMINAL CARE: CAN THE KARNOFSKY INDEX HELP? , 1985, The Lancet.

[3]  J. Lynn,et al.  Predicting life span for applicants to inpatient hospice. , 1988, Archives of internal medicine.

[4]  F. Harrell,et al.  Predicting outcome in coronary disease. Statistical models versus expert clinicians. , 1986, The American journal of medicine.

[5]  A S Detsky,et al.  Prognosis, survival, and the expenditure of hospital resources for patients in an intensive-care unit. , 1981, The New England journal of medicine.

[6]  D. Bamber.  The area above the ordinal dominance graph and the area below the receiver operating characteristic graph , 1975 .

[7]  P. Nowell.  Mechanisms of tumor progression. , 1986, Cancer research.

[8]  W. Knaus,et al.  Short-term mortality predictions for critically ill hospitalized adults: science and ethics. , 1991, Science.

[9]  P. Greenland,et al.  Selection and interpretation of diagnostic tests and procedures. Principles and applications. , 1981, Annals of internal medicine.

[10]  W. Mackillop,et al.  Cancer patients' perceptions of their disease and its treatment. , 1988, British Journal of Cancer.

[11]  J. Swets.  ROC analysis applied to the evaluation of medical imaging techniques. , 1979, Investigative radiology.

[12]  R M Poses,et al.  The answer to “What are my chances, Doctor?” depends on whom is asked: Prognostic disagreement and inaccuracy for critically ill patients , 1989, Critical care medicine.

[13]  A L Brannen,et al.  Prediction of outcome from critical illness. A comparison of clinical judgment with a prediction rule. , 1989, Archives of internal medicine.

[14]  M A Hlatky,et al.  Clinical experience and predicting survival in coronary disease. , 1989, Archives of internal medicine.

[15]  Roy M. Poses,et al.  Availability, Wishful Thinking, and Physicians' Diagnostic Judgments for Patients with Suspected Bacteremia , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[16]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[17]  Elizabeth C. Hirschman,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[18]  G. Brier,et al.  External correspondence: Decompositions of the mean probability score , 1982 .

[19]  A. Feinstein,et al.  SYMPTOMATIC PATTERNS, BIOLOGIC BEHAVIOR, AND PROGNOSIS IN CANCER OF THE LUNG. PRACTICAL APPLICATION OF BOOLEAN ALGEBRA AND CLINICAL TAXONOMY. , 1964, Annals of internal medicine.

[20]  H. Skipper,et al.  EXPERIMENTAL EVALUATION OF POTENTIAL ANTICANCER AGENTS. XIII. ON THE CRITERIA AND KINETICS ASSOCIATED WITH "CURABILITY" OF EXPERIMENTAL LEUKEMIA. , 1964, Cancer chemotherapy reports.

[21]  V. Devita,et al.  Cancer : Principles and Practice of Oncology , 1982 .

[22]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[23]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[24]  D. McClish,et al.  How Well Can Physicians Estimate Mortality in a Medical Intensive Care Unit? , 1989, Medical decision making : an international journal of the Society for Medical Decision Making.

[25]  J. Habbema,et al.  Prognosis in medicine: An analysis of its meaning and rôles , 1987, Theoretical medicine.

[26]  Parkes Cm,et al.  Accuracy of predictions of survival in later stages of cancer. , 1972 .

[27]  R M Centor,et al.  Eualuating Physicians' Probabilistic Judgments , 1988, Medical decision making : an international journal of the Society for Medical Decision Making.

[28]  Baruch Fischhoff,et al.  Calibration of Probabilities: The State of the Art , 1977 .

[29]  J. Hilden,et al.  Single Testing When Two Tests Are Available To the Editor , 1988 .