Finding profitable forecast combinations using probability scoring rules

This study examines the success of bets on Australian Football League (AFL) matches made by identifying panels of highly proficient forecasters and betting on the basis of their pooled opinions. The data set is unusual, in that all forecasts are in the form of probabilities. Bets are made "on paper"Â against quoted market betting odds according to the (fractional) Kelly criterion. To identify expertise, individual forecasters are scored using conventional probability scoring rules, a "Kelly score"Â representing the forecaster's historical paper profits from Kelly-betting, and the more simplistic "categorical score"Â (number of misclassifications). Despite implicitly truncating all probabilities to either 0 or 1 before evaluation, and thus losing a lot of information, the categorical scoring rule appears to be a propitious way of ranking probability forecasters. Bootstrap significance tests indicate that this improvement is not attributable to chance.

[1]  C. Granger,et al.  Handbook of Economic Forecasting , 2006 .

[2]  R. L. Winkler.  The Quantification of Judgment: Some Methodological Suggestions , 1967 .

[3]  A. Timmermann.  Chapter 4 Forecast Combinations , 2006 .

[4]  R. Clemen.  Combining forecasts: A review and annotated bibliography , 1989 .

[5]  J. J. Kelly.  A new interpretation of information rate , 1956 .

[6]  E. Thorp.  The Kelly Criterion in Blackjack Sports Betting, and the Stock Market , 2008 .

[7]  A. H. Murphy,et al.  Screening probability forecasts: contrasts between choosing and combining , 1995 .

[8]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[9]  Jose A. Lopez.  Evaluating the Predictive Accuracy of Volatility Models , 2001 .

[10]  L. J. Savage.  Elicitation of Personal Probabilities and Expectations , 1971 .

[11]  David J. Johnstone,et al.  The Parimutuel Kelly Probability Scoring Rule , 2007, Decis. Anal..

[12]  W. Ziemba,et al.  Capital growth with security , 2004 .

[13]  C. Granger,et al.  Economic and Statistical Measures of Forecast Accuracy , 1999 .

[14]  W. Ziemba,et al.  Growth versus security in dynamic investment analysis , 1992 .

[15]  A. H. Murphy,et al.  “Good” Probability Assessors , 1968 .

[16]  R. L. Winkler,et al.  Combining Economic Forecasts , 1986 .

[17]  Herman Stekler,et al.  Are economic forecasts valuable , 1994 .

[18]  David Johnstone,et al.  Economic Darwinism: Who has the Best Probabilities? , 2007 .

[19]  Robert L. Winkler,et al.  Expert resolution , 1986 .

[20]  R. L. Winkler,et al.  Separating probability elicitation from utilities , 1988 .

[21]  Roy Batchelor,et al.  Forecaster diversity and the benefits of combining forecasts , 1995 .

[22]  William T. Ziemba,et al.  Growth versus security tradeoffs indynamic investment analysis , 1999, Ann. Oper. Res..

[23]  Robert L. Winkler,et al.  Probability Elicitation, Scoring Rules, and Competition Among Forecasters , 2007, Manag. Sci..

[24]  W. Poundstone.  Fortune's Formula: The Untold Story of the Scientific Betting System That Beat the Casinos and Wall Street , 2005 .

[25]  Robert L. Winkler,et al.  Simple robust averages of forecasts: Some empirical results , 2008 .

[26]  E. Thorp.  OPTIMAL GAMBLING SYSTEMS FOR FAVORABLE GAMES , 1969 .

[27]  Steven D. Levitt,et al.  Why are Gambling Markets Organized so Differently from Financial Markets? , 2004 .

[28]  J. C. Thompson,et al.  THE ECONOMIC UTILITY OF WEATHER FORECASTS , 1955 .

[29]  H. White,et al.  Data‐Snooping, Technical Trading Rule Performance, and the Bootstrap , 1999 .

[30]  H. White,et al.  A Reality Check for Data Snooping , 2000 .

[31]  William R. Eadington,et al.  Finding the edge : mathematical analysis of casino games , 2000 .

[32]  Robert L. Winkler,et al.  Probabilistic Prediction: Some Experimental Results , 1971 .

[33]  Robert L. Winkler,et al.  Multiple Experts vs. Multiple Methods: Combining Correlation Assessments , 2004, Decis. Anal..

[34]  Ray C. Fair,et al.  Evaluating the predictive accuracy of models , 1986 .

[35]  Zizhuo Wang,et al.  A unified framework for dynamic pari-mutuel information market design , 2009, EC '09.

[36]  Michael P. Clements,et al.  Evaluating the Bank of England Density Forecasts of Inflation , 2004 .

[37]  A. Diederich,et al.  Evaluating and Combining Subjective Probability Estimates , 1997 .

[38]  Bruno de Finetti,et al.  Logical foundations and measurement of subjective probability , 1970 .

[39]  David Lindley.  Scoring rules and the inevitability of probability , 1982 .

[40]  A. H. Murphy,et al.  Scoring rules and the evaluation of probabilities , 1996 .

[41]  David M. Pennock,et al.  Information markets vs. opinion pools: an empirical comparison , 2005, EC '05.

[42]  Robert L. Winkler,et al.  Combining Probability Distributions From Experts in Risk Analysis , 1999 .

[43]  Yuming Li.  Growth-security investment strategy for long and short runs , 1993 .

[44]  David M. Pennock,et al.  Prediction Markets: Does Money Matter? , 2004, Electron. Mark..

[45]  J. Graham.  Is a Group of Economists Better than One? Than None? , 1996 .