Bayesian Ensembles of Binary-Event Forecasts: When Is It Appropriate to Extremize or Anti-Extremize?

Many organizations face critical decisions that rely on forecasts of binary events---events such as whether a borrower will default on a loan or not. In these situations, organizations often gather forecasts from multiple experts or models. This raises the question of how to aggregate the forecasts. Because linear combinations of probability forecasts are known to be underconfident, we introduce a class of aggregation rules, or Bayesian ensembles, that are non-linear in the experts' probabilities. These ensembles are generalized additive models of experts' probabilities. These models have three key properties. They are coherent, i.e., consistent with the Bayesian view. They can aggregate calibrated (or miscalibrated) forecasts. And they are often more extreme, and therefore more confident, than the commonly used linear opinion pool. Empirically, we demonstrate that our ensemble can be easily fit to real data using a generalized linear modeling framework. We use this framework to aggregate several forecasts of binary events in two publicly available datasets. The forecasts come from several leading statistical and machine learning algorithms. Our Bayesian ensemble offers an improvement out-of-sample over the linear opinion pool and over any one of the individual algorithms considered.