Non-asymptotic calibration and resolution

We analyze a new algorithm for probability forecasting of binary labels, without making any assumptions about the way the data is generated. The algorithm is shown to be well calibrated and to have high resolution for big enough data sets and for a suitable choice of its parameter, a kernel on the Cartesian product of the forecast space [0,1] and the object space. Our results are non-asymptotic: we establish explicit inequalities for the performance of the algorithm.

[1]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[2]  G. Wahba.  Smoothing noisy data with spline functions , 1975 .

[3]  J. B. Roberts,et al.  Elements of the theory of functions , 1967 .

[4]  R. Varga,et al.  Proof of Theorem 1 , 1983 .

[5]  Don R. Hush,et al.  Function Classes That Approximate the Bayes Risk , 2006, COLT.

[6]  M. Duflo,et al.  Sur la loi des grands nombres pour les martingales vectorielles et l'estimateur des moindres carrés d'un modèle de régression , 1990 .

[7]  A. Dawid.  Self-Calibrating Priors Do Not Exist: Comment , 1985 .

[8]  Akimichi Takemura,et al.  Defensive Forecasting for Linear Protocols , 2005, ALT.

[9]  Vladimir Vovk.  Defensive Prediction with Expert Advice , 2005, ALT.

[10]  William H. Press,et al.  Numerical recipes in C , 2002 .

[11]  Mark J. Schervish,et al.  Discussion: Calibration-Based Empirical Probability , 1985 .

[12]  Alvaro Sandroni,et al.  Calibration with Many Checking Rules , 2003, Math. Oper. Res..

[13]  David Haussler,et al.  How to use expert advice , 1993, STOC.

[14]  Vladimir Vovk,et al.  On-Line Regression Competitive with Reproducing Kernel Hilbert Spaces , 2005, TAMC.

[15]  Vladimir Vovk,et al.  Predictions as Statements and Decisions , 2006, COLT.

[16]  E. Lehrer.  Any Inspection is Manipulable , 2001 .

[17]  Herbert Meschkowski,et al.  Hilbertsche Räume mit Kernfunktion , 1962 .

[18]  A Cordoba,et al.  On differentiation of integrals. , 1977, Proceedings of the National Academy of Sciences of the United States of America.

[19]  Grace Wahba,et al.  Spline Models for Observational Data , 1990 .

[20]  A. Berlinet,et al.  Reproducing kernel Hilbert spaces in probability and statistics , 2004 .

[21]  Par N. Aronszajn.  La théorie des noyaux reproduisants et ses applications Première Partie , 1943, Mathematical Proceedings of the Cambridge Philosophical Society.

[22]  Sham M. Kakade,et al.  Deterministic calibration and Nash equilibrium , 2004, J. Comput. Syst. Sci..

[23]  Alvaro Sandroni,et al.  The reproducible properties of correct forecasts , 2003, Int. J. Game Theory.

[24]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[25]  Jean-Luc Ville.  Étude critique de la notion de collectif , 1939 .

[26]  W. Stout.  A martingale analogue of Kolmogorov's law of the iterated logarithm , 1970 .

[27]  Vladimir Vovk.  Non-asymptotic calibration and resolution , 2007, Theor. Comput. Sci..

[28]  J. Lewins.  Contribution to the Discussion , 1989 .

[29]  G. Wahba.  Spline models for observational data , 1990 .

[30]  David Oakes,et al.  Self-Calibrating Priors Do Not Exist , 1985 .

[31]  G. Shafer,et al.  Probability and Finance: It's Only a Game! , 2001 .

[32]  Alexander J. Smola,et al.  Learning with kernels , 1998 .

[33]  Akimichi Takemura,et al.  Defensive Forecasting , 2005, AISTATS.

[34]  Vladimir Vovk.  Competitive on-line learning with a convex loss function , 2005, ArXiv.

[35]  Ingo Steinwart,et al.  On the Influence of the Kernel on the Consistency of Support Vector Machines , 2002, J. Mach. Learn. Res..

[36]  Marianna Csörnyei.  Absolutely Continuous Functions of Rado, Reichelderfer, and Malý☆ , 2000 .

[37]  J. Doob.  Stochastic processes , 1953 .

[38]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[39]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[40]  G. Shafer,et al.  Good randomized sequential probability forecasting is always possible , 2005 .