Complexity-Based Approach to Calibration with Checking Rules

We consider the problem of forecasting a sequence of outcomes from an unknown source. The quality of the forecaster is measured by a family of checking rules. We prove upper bounds on the value of the associated game, thus certifying the existence of a calibrated strategy for the forecaster. We show that complexity of the family of checking rules can be captured by the notion of a sequential cover introduced in (Rakhlin et al., 2010a). Various natural assumptions on the class of checking rules are considered, including niteness of Vapnik-Chervonenkis and Littlestone’s dimensions.

[1]  John Langford,et al.  Correlated equilibria in graphical games , 2003, EC '03.

[2]  Peter L. Bartlett,et al.  Blackwell Approachability and No-Regret Learning are Equivalent , 2010, COLT.

[3]  Shai Ben-David,et al.  Agnostic Online Learning , 2009, COLT.

[4]  R. Vohra,et al.  Calibrated Learning and Correlated Equilibrium , 1996 .

[5]  N. Littlestone Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[6]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[7]  Ambuj Tewari,et al.  Online Learning: Random Averages, Combinatorial Parameters, and Learnability , 2010, NIPS.

[8]  I. Pinelis OPTIMUM BOUNDS FOR THE DISTRIBUTIONS OF MARTINGALES IN BANACH SPACES , 1994, 1208.2200.

[9]  Ambuj Tewari,et al.  Online Learning: Beyond Regret , 2010, COLT.

[10]  David Oakes,et al.  Self-Calibrating Priors Do Not Exist , 1985 .

[11]  Sham M. Kakade,et al.  Deterministic calibration and Nash equilibrium , 2004, J. Comput. Syst. Sci..

[12]  Alvaro Sandroni,et al.  A nonmanipulable test , 2009, 0904.0338.

[13]  Paul W. Goldberg,et al.  Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers , 1993, COLT '93.

[14]  Ehud Lehrer The Game of Normal Numbers , 2004, Math. Oper. Res..

[15]  E. Giné,et al.  Decoupling: From Dependence to Independence , 1998 .

[16]  E. Kalai,et al.  Calibrated Forecasting and Merging , 1999 .

[17]  H. Peyton Young,et al.  Strategic Learning and Its Limits , 2004 .

[18]  Alvaro Sandroni,et al.  Calibration with Many Checking Rules , 2003, Math. Oper. Res..

[19]  Shie Mannor,et al.  A Geometric Proof of Calibration , 2009, Math. Oper. Res..

[20]  A. Dawid The Well-Calibrated Bayesian , 1982 .