Combining multiple probability predictions using a simple logit model

This paper begins by presenting a simple model of the way in which experts estimate probabilities. The model is then used to construct a likelihood-based aggregation formula for combining multiple probability forecasts. The resulting aggregator has a simple analytical form that depends on a single, easily-interpretable parameter. This makes it computationally simple, attractive for further development, and robust against overfitting. Based on a large-scale dataset in which over 1300 experts tried to predict 69 geopolitical events, our aggregator is found to be superior to several widely-used aggregation algorithms.

[1]  A. Raftery,et al.  Probabilistic forecasts, calibration and sharpness , 2007 .

[2]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[3]  H Gu,et al.  The effects of averaging subjective probability estimates between and within judges. , 2000, Journal of experimental psychology. Applied.

[4]  Andreas Graefe,et al.  Combining Forecasts: An Application to Elections , 2013 .

[5]  Lisa Werner,et al.  Principles of forecasting: A handbook for researchers and practitioners , 2002 .

[6]  T. Gneiting,et al.  Combining probability forecasts , 2010 .

[7]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[8]  Hang Zhang,et al.  Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition , 2012, Front. Neurosci..

[9]  A. Diederich,et al.  Evaluating and Combining Subjective Probability Estimates , 1997 .

[10]  E. I. Polyakova,et al.  The Nu Expression for Probabilistic Data Integration , 2007 .

[11]  A dynamic oligopoly model with demand inertia and inventories , 1989 .

[12]  Soumik Pal,et al.  A note on a conjectured sharpness principle for probabilistic forecasting with calibration , 2009, 0902.0342.

[13]  J. Armstrong,et al.  PRINCIPLES OF FORECASTING 1 Principles of Forecasting : A Handbook for Researchers and Practitioners , 2006 .

[14]  G. Brier.  VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[15]  Ilan Yaniv,et al.  Overconfidence in interval estimates: What does expertise buy you? , 2008 .

[16]  Adele Diederich,et al.  Understanding pooled subjective probability estimates , 2001, Math. Soc. Sci..

[17]  Jonathan Baron,et al.  Two Reasons to Make Aggregated Probability Forecasts More Extreme , 2014, Decis. Anal..

[18]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[19]  Roopesh Ranjan.  Combining and evaluating probabilistic forecasts , 2009 .

[20]  Frank Lad,et al.  Learning from the Probability Assertions of Experts , 2003 .

[21]  A. Buja,et al.  Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications , 2005 .

[22]  Craig R. Fox,et al.  Partition Priming in Judgment Under Uncertainty , 2003, Psychological science.

[23]  I. Erev,et al.  Simultaneous Over- and Underconfidence: The Role of Error in Judgment Processes. , 1994 .

[24]  Christian Genest,et al.  Combining Probability Distributions: A Critique and an Annotated Bibliography , 1986 .

[25]  R. Bordley.  A Multiplicative Formula for Aggregating Probability Assessments , 1982 .

[26]  P. Renard,et al.  Probability Aggregation Methods in Geoscience , 2012, Mathematical Geosciences.