The Aggregative Contingent Estimation System: Selecting, Rewarding, and Training Experts in a Wisdom of Crowds Approach to Forecasting

We describe the Aggregative Contingent Estimation System (http://www.forecastingace.com), which is designed to elicit and aggregate forecasts from large, diverse groups of individuals. The Aggregative Contingent Estimation System (ACES; see http://www.forecastingace.com) is a project funded by the Intelligence Advanced Research Projects Activity. The project, which is a collaboration between seven universities and a private company (Applied Research Associates), utilizes a crowdsourcing approach to forecast global events such as the outcome of presidential elections in Taiwan and the potential of a downgrade of Greek sovereign debt. The main project goal is to develop new methods for collecting and combining forecasts of many widely-dispersed individuals in order to increase aggregated forecasts’ predictive accuracy. A future goal of this project will involve the development of methods for effectively communicating forecast results to decision makers, the end users of the forecasts. To test our methods, we are engaging members of the general public to voluntarily provide web-based forecasts at their convenience. Our engagement of the general public in this endeavor has brought up a host of issues that involve translation of basic research to the applied problem of global forecasting. In this case study, we focus on three aspects of the project that have general crowdsourcing implications: strategies for rewarding the contributors, strategies for training contributors to be better forecasters, and methods for selecting experts (i.e., estimating the extent to which one is an expert for the purpose of weighting forecasts). We also provide an overview of our statistical aggregation models that are consistently beating the baseline forecasts (the unweighted average forecasts).