Prediction with Expert Evaluators' Advice

We introduce a new protocol for prediction with expert advice in which each expert evaluates the learner's and his own performance using a loss function that may change over time and may be different from the loss functions used by the other experts. The learner's goal is to perform better or not much worse than each expert, as evaluated by that expert, for all experts simultaneously. If the loss functions used by the experts are all proper scoring rules and all mixable, we show that the defensive forecasting algorithm enjoys the same performance guarantee as that attainable by the Aggregating Algorithm in the standard setting and known to be optimal. This result is also applied to the case of "specialist" experts. In this case, the defensive forecasting algorithm reduces to a simple modification of the Aggregating Algorithm.

[1]  Yoram Singer,et al.  Using and combining predictors that specialize , 1997, STOC '97.

[2]  A. Dawid,et al.  Prequential probability: principles and properties , 1999 .

[3]  David Haussler,et al.  Sequential Prediction of Individual Sequences Under General Loss Functions , 1998, IEEE Trans. Inf. Theory.

[4]  Akimichi Takemura,et al.  Defensive Forecasting , 2005, AISTATS.

[5]  Yishay Mansour,et al.  From External to Internal Regret , 2005, J. Mach. Learn. Res..

[6]  Vladimir Vovk,et al.  Defensive forecasting for optimal prediction with expert advice , 2007, ArXiv.

[7]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[8]  A. Buja,et al.  Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications , 2005 .

[9]  Yoram Singer,et al.  Context-sensitive learning methods for text categorization , 1996, SIGIR '96.

[10]  William H. Press,et al.  Numerical recipes in C , 2002 .

[11]  A. Raftery,et al.  Strictly Proper Scoring Rules, Prediction, and Estimation , 2007 .

[12]  Robert D. Kleinberg,et al.  Regret bounds for sleeping experts and bandits , 2010, Machine Learning.

[13]  William H. Press,et al.  Numerical Recipes in C, 2nd Edition , 1992 .

[14]  Vladimir Vovk,et al.  Derandomizing Stochastic Prediction Strategies , 1997, COLT '97.

[15]  Vladimir Vovk,et al.  Aggregating strategies , 1990, COLT '90.

[16]  Vladimir Vovk,et al.  A game of prediction with expert advice , 1995, COLT '95.

[17]  Vladimir Vovk,et al.  Supermartingales in prediction with expert advice , 2008, Theor. Comput. Sci..

[18]  G. Shafer,et al.  Probability and Finance: It's Only a Game! , 2001 .