How do surgeons' probability estimates of operative mortality compare with a decision analytic model?

The aim of this study is to compare surgeons' estimates of operative mortality of patients with an abdominal aneurysm (= dilation of the aorta) with the operative mortality derived from a decision analytic model and to determine how surgeons use clinical information. Four experienced surgeons are asked to estimate, among other things, the operative mortality of 137 patients. Results concerning the accuracy of surgeons' estimates show that surgeons' average operative mortality estimates are quite accurate as compared to the calculated mortalities. The standard deviations of surgeons' estimates are lower than the standard deviation of the model, however, indicating that the surgeons are not as good in distinguishing the high and low risk patients. Furthermore, surgeons show substantial inconsistencies in the weighing of the clinical information, and also differ from the model in how clinical information is weighed. Finally, when comparing the operative mortalities of the patients who died and those who did not, the model shows a modest, but higher discrimination than the surgeons. Physicians' performance seems to be influenced by the difficulty of the task (i.e. the unpredictability of the event and the multidimensionality of the task). In order to improve physicians' probability estimates, the calculations of the decision model can be used as learning tool.

[1]  J A Swets,et al.  Enhancing and Evaluating Diagnostic Accuracy , 1991, Medical decision making : an international journal of the Society for Medical Decision Making.

[2]  A S Detsky,et al.  Prognosis, survival, and the expenditure of hospital resources for patients in an intensive-care unit. , 1981, The New England journal of medicine.

[3]  B. Brundage,et al.  Diagnostic accuracy of cardiologists compared with probability calculations using Bayes' rule. , 1982, The American journal of cardiology.

[4]  J C Hershey,et al.  Assessing Physicians Estimates of the Probability of Coronary Artery Disease , 1992, Medical decision making : an international journal of the Society for Medical Decision Making.

[5]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[6]  J. Yates,et al.  How quantity judgment changes as the number of cues increases: An analytical framework and review. , 1992 .

[7]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[8]  Robert L. Winkler,et al.  An Assessment of the Risk of Chronic Lung Injury Attributable to Long-Term Ozone Exposure , 1995, Oper. Res..

[9]  P. Powell.  Expertise and Decision Support , 2013 .

[10]  Robert L. Winkler,et al.  Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit , 1993 .

[11]  B. Fischhoff,et al.  Training for calibration. , 1980 .

[12]  Jay J.J. Christensen-Szalanski,et al.  Physicians' use of probabilistic information in a real clinical setting. , 1981 .

[13]  T. Wallsten.  Physician and Medical Student Bias in Evaluating Diagnostic Information , 1981, Medical decision making : an international journal of the Society for Medical Decision Making.

[14]  M Bobbio,et al.  Clinical Assessment of the Probability of Coronary Artery Disease , 1992, Medical decision making : an international journal of the Society for Medical Decision Making.

[15]  R D Cebul,et al.  The accuracy of experienced physicians' probability estimates for patients with sore throats. Implications for decision making. , 1985, JAMA.

[16]  A. H. Murphy,et al.  Probability Forecasting in Meteorology , 1984 .

[17]  J. Frank Yates,et al.  Subjective probability accuracy analysis. , 1994 .

[18]  Robert R. Hoffman,et al.  The Psychology of Expertise , 1925 .

[19]  D. McClish,et al.  How Well Can Physicians Estimate Mortality in a Medical Intensive Care Unit? , 1989, Medical decision making : an international journal of the Society for Medical Decision Making.

[20]  George Wright,et al.  Coherence, Calibration, and Expertise in Judgmental Probability Forecasting , 1994 .

[21]  J. Habbema,et al.  Perioperative mortality of elective abdominal aortic aneurysm surgery. A clinical prediction rule based on literature and individual patient data. , 1995, Archives of internal medicine.

[22]  F. Harrell,et al.  Predicting outcome in coronary disease. Statistical models versus expert clinicians. , 1986, The American journal of medicine.

[23]  W M Tierney,et al.  Physicians' Estimates of the Probability of Myocardial Infarction in Emergency Boom Patients with chest Pain , 1986, Medical decision making : an international journal of the Society for Medical Decision Making.

[24]  M A Hlatky,et al.  Clinical experience and predicting survival in coronary disease. , 1989, Archives of internal medicine.

[25]  R. Wigton,et al.  Relation of Physicians' Predicted Probabilities of Pneumonia to Their Utilities for Ordering Chest X-rays to Detect Pneumonia , 1992, Medical decision making : an international journal of the Society for Medical Decision Making.