Non-asymptotic calibration and resolution

We analyze a new algorithm for probability forecasting of binary observations on the basis of the available data, without making any assumptions about the way the observations are generated. The algorithm is shown to be well-calibrated and to have good resolution for long enough sequences of observations and for a suitable choice of its parameter, a kernel on the Cartesian product of the forecast space [0, 1] and the data space. Our main results are non-asymptotic: we establish explicit inequalities, shown to be tight, for the performance of the algorithm.

[1]  Alvaro Sandroni,et al.  The reproducible properties of correct forecasts , 2003, Int. J. Game Theory.

[2]  G. Shafer,et al.  Good randomized sequential probability forecasting is always possible , 2005 .

[3]  Akimichi Takemura,et al.  Defensive Forecasting , 2005, AISTATS.

[4]  Par N. Aronszajn.  La théorie des noyaux reproduisants et ses applications Première Partie , 1943, Mathematical Proceedings of the Cambridge Philosophical Society.

[5]  Sham M. Kakade,et al.  Deterministic calibration and Nash equilibrium , 2004, Journal of computer and system sciences (Print).

[6]  A. Dawid.  Comment: The Impossibility of Inductive Inference , 1985 .

[7]  G. Shafer,et al.  Probability and Finance: It's Only a Game! , 2001 .

[8]  Patrick Brézillon,et al.  Lecture Notes in Artificial Intelligence , 1999 .

[9]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[10]  Vladimir Vovk,et al.  Predictions as Statements and Decisions , 2006, COLT.

[11]  E. Lehrer.  Any Inspection is Manipulable , 2001 .

[12]  Herbert Meschkowski,et al.  Hilbertsche Räume mit Kernfunktion , 1962 .

[13]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[14]  A. Dawid.  Self-Calibrating Priors Do Not Exist: Comment , 1985 .

[15]  Alvaro Sandroni,et al.  Calibration with Many Checking Rules , 2003, Math. Oper. Res..

[16]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[17]  A. Dawid.  Calibration-Based Empirical Probability , 1985 .

[18]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[19]  Jean-Luc Ville.  Étude critique de la notion de collectif , 1939 .

[20]  William H. Press,et al.  Numerical recipes in C++: the art of scientific computing, 2nd Edition (C++ ed., print. is corrected to software version 2.10) , 1994 .

[21]  J. Suchorzewska.  Głos w dyskusji , 2005 .

[22]  Mark J. Schervish,et al.  Discussion: Calibration-Based Empirical Probability , 1985 .

[23]  David Oakes,et al.  Self-Calibrating Priors Do Not Exist , 1985 .

[24]  W. Stout.  A martingale analogue of Kolmogorov's law of the iterated logarithm , 1970 .

[25]  William H. Press,et al.  Numerical recipes in C , 2002 .

[26]  J. Doob.  Stochastic processes , 1953 .

[27]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[28]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[29]  Don R. Hush,et al.  Function Classes That Approximate the Bayes Risk , 2006, COLT.

[30]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[31]  G. Wahba.  Smoothing noisy data with spline functions , 1975 .

[32]  Mark J. Schervish,et al.  Self-Calibrating Priors Do Not Exist: Comment , 1985 .

[33]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[34]  Vladimir Vovk.  Defensive Prediction with Expert Advice , 2005, ALT.