Testing Exchangeability On-Line

The majority of theoretical work in machine learning is done under the assumption of exchangeability: essentially, it is assumed that the examples are generated from the same probability distribution independently. This paper is concerned with the problem of testing the exchangeability assumption in the on-line mode: examples are observed one by one and the goal is to monitor on-line the strength of evidence against the hypothesis of exchangeability. We introduce the notion of exchangeability martingales, which are online procedures for detecting deviations from exchangeability; in essence, they are betting schemes that never risk bankruptcy and are fair under the hypothesis of exchangeability. Some specific exchangeability martingales are constructed using Transductive Confidence Machine. We report experimental results showing their performance on the USPS benchmark data set of hand-written digits (known to be somewhat heterogeneous); one of them multiplies the initial capital by more than 1018; this means that the hypothesis of exchangeability is rejected at the significance level 10-18.

[1]  Vladimir Vovk,et al.  A Universal Well-Calibrated Algorithm for On-line Classification , 2004, J. Mach. Learn. Res..

[2]  Vladimir Vovk,et al.  On-line confidence machines are well-calibrated , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[3]  Vladimir Vovk,et al.  Aggregating strategies , 1990, COLT '90.

[4]  Sheldon M. Ross,et al.  Stochastic Processes , 2018, Gauge Integral Structures for Stochastic Calculus and Quantum Electrodynamics.

[5]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[6]  G. Shafer,et al.  Probability and Finance: It's Only a Game! , 2001 .

[7]  F. Y. Edgeworth,et al.  The theory of statistics , 1996 .

[8]  Vladimir Vovk,et al.  Asymptotic Optimality of Transductive Confidence Machine , 2002, ALT.

[9]  Vladimir Vovk,et al.  Online Region Prediction with Real Teachers , 2003 .

[10]  Alexander Gammerman,et al.  Machine-Learning Applications of Algorithmic Randomness , 1999, ICML.

[11]  Mark Herbster,et al.  Tracking the Best Expert , 1995, Machine Learning.

[12]  V. Vovk A logic of probability, with application to the foundations of statistics , 1993 .

[13]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[14]  Vladimir Vovk,et al.  Derandomizing Stochastic Prediction Strategies , 1997, COLT '97.