Robust forecast aggregation

Significance The problem of conflicting advice from multiple experts is common to policy makers, corporate governors, patients facing medical prognosis, individuals requiring financial advice, and more. We ask, How should an ignorant advisee (one that observes experts’ forecasts only) aggregate information when this is provided in probabilistic terms (such as forecasts over events)? We propose a robustness criterion based on the classical notions of scoring rule and regret. Under reasonable assumptions on the underlying information structure of the experts, we provide formulas that allow an ignorant aggregator to perform almost as well as an omniscient expert (one that aggregates perfectly all of the information) whenever there are two experts. We also show that this is hopeless when facing many experts. Bayesian experts who are exposed to different evidence often make contradictory probabilistic forecasts. An aggregator, ignorant of the underlying model, uses this to calculate his or her own forecast. We use the notions of scoring rules and regret to propose a natural way to evaluate an aggregation scheme. We focus on a binary state space and construct low regret aggregation schemes whenever there are only two experts that either are Blackwell-ordered or receive conditionally independent and identically distributed (i.i.d.) signals. In contrast, if there are many experts with conditionally i.i.d. signals, then no scheme performs (asymptotically) better than a (0.5,0.5) forecast.

[1]  R. L. Winkler.  Combining Probability Distributions from Dependent Information Sources , 1981 .

[2]  Jérôme Renault,et al.  Repeated Games with Incomplete Information , 2009, Encyclopedia of Complexity and Systems Science.

[3]  H. Sebastian Seung,et al.  A solution to the single-question crowd wisdom problem , 2017, Nature.

[4]  Yakov Babichenko,et al.  When is the Crowd Wise? , 2017 .

[5]  Jonathan Baron,et al.  Combining multiple probability predictions using a simple logit model , 2014 .

[6]  A. Timmermann.  Chapter 4 Forecast Combinations , 2006 .

[7]  Victor Zarnowitz,et al.  The Accuracy of Individual and Group Forecasts from Business Outlook Surveys , 1982 .

[8]  Yakov Babichenko,et al.  Learning of Optimal Forecast Aggregation in Partial Evidence Environments , 2018, Math. Oper. Res..

[9]  R. Bordley.  A Multiplicative Formula for Aggregating Probability Assessments , 1982 .

[10]  C. Granger,et al.  Experience with Forecasting Univariate Time Series and the Combination of Forecasts , 1974 .

[11]  Arnold Zellner,et al.  "Bayesian and Non-Bayesian Methods for Combining Models and Forecasts with Applications to Forecasting International Growth Rates" , 2004 .

[12]  Robin Pemantle,et al.  Bayesian aggregation of two forecasts in the partial information framework , 2016, 1608.04717.

[13]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[14]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[15]  Antonello D’Agostino,et al.  Understanding and forecasting aggregate and disaggregate price dynamics , 2011, SSRN Electronic Journal.

[16]  Colin Stewart,et al.  Testing Multiple Forecasters , 2007 .

[17]  M. Degroot.  Reaching a Consensus , 1974 .

[18]  J. Stock,et al.  Combination forecasts of output growth in a seven-country data set , 2004 .

[19]  M. Sion.  On general minimax theorems , 1958 .

[20]  Manuel Mueller-Frank,et al.  A general framework for rational learning in social networks , 2011 .

[21]  D. Blackwell.  Equivalent Comparisons of Experiments , 1953 .

[22]  Lyle H. Ungar,et al.  Modeling Probability Forecasts via Information Diversity , 2014 .

[23]  P. Bickel.  Parametric Robustness: Small Biases can be Worthwhile , 1984 .

[24]  Arnold Zellner,et al.  Bayesian and non-Bayesian methods for combining models and forecasts with applications to forecasting international growth rates , 1993 .

[25]  Alvaro Sandroni.  Do markets favor agents able to make accurate predictions , 2000 .

[26]  I. Gilboa,et al.  Maxmin Expected Utility with Non-Unique Prior , 1989 .

[27]  Nabil I. Al-Najjar,et al.  Comparative Testing of Experts , 2006 .

[28]  James Hannan,et al.  4. APPROXIMATION TO RAYES RISK IN REPEATED PLAY , 1958 .

[29]  Alvaro Sandroni,et al.  Manipulability of comparative tests , 2009, Proceedings of the National Academy of Sciences.

[30]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .