Does the expert know? The reliability of predictions and confidence ratings of experts

Experts are often asked to assess two different kinds of probabilities. One is THE PROBABILITY THAT SOMETHING WILL HAPPEN: rain, hitting an oil well, dying in an operation, tube fracture, a total melt-down. We will call these probabilities “predictions”. Experts’ predictions are widely used in all sorts of personal and public decision making. Two examples of formal usage of expert opinion are risk analyses and expert systems used for diagnostic tasks. The second kind of probability assessed by experts is THE PROBABILITY THAT THEIR ANSWERS ARE CORRECT. We will call these probabilities “confidence ratings”. A little later we will demonstrate that predictions and confidence ratings have often been confused in the literature.