Conformal testing in a binary model situation

Conformal testing is a way of testing the IID assumption based on conformal prediction. The topic of this note is computational evaluation of the performance of conformal testing in a model situation in which IID binary observations generated from a Bernoulli distribution are followed by IID binary observations generated from another Bernoulli distribution, with the parameters of the distributions and changepoint unknown. Existing conformal test martingales can be used for this task and work well in simple cases, but their efficiency can be improved greatly.

[1]  E. S. Pearson,et al.  On the Problem of the Most Efficient Tests of Statistical Hypotheses , 1933 .

[2]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[3]  Alexander Gammerman,et al.  Plug-in martingales for testing exchangeability on-line , 2012, ICML.

[4]  Vladimir Vovk,et al.  Testing Randomness Online , 2019, Statistical Science.

[5]  Andrew R. Barron,et al.  Asymptotic minimax regret for data compression, gambling, and prediction , 1997, IEEE Trans. Inf. Theory.

[6]  Jorma Rissanen,et al.  Minimum Description Length Principle , 2010, Encyclopedia of Machine Learning.

[7]  J. Wolfowitz,et al.  Optimum Character of the Sequential Probability Ratio Test , 1948 .

[8]  Wouter M. Koolen,et al.  How can one test if a binary sequence is exchangeable? Fork-convex hulls, supermartingales, and Snell envelopes , 2021, ArXiv.

[9]  G. Shafer The Language of Betting as a Strategy for Statistical and Scientific Communication , 2019, 1903.06991.

[10]  V. Vovk On the concept of the Bernoulli property , 1986 .

[11]  Conformal changepoint detection in continuous model situations , 2021 .

[12]  J. Walsh A property of conformal martingales , 1977 .

[13]  Vladimir Vovk,et al.  Testing for concept shift online , 2020, ArXiv.

[14]  R. Getoor,et al.  Conformal martingales , 1972 .

[15]  Vladimir Vovk,et al.  Derandomizing Stochastic Prediction Strategies , 1997, COLT '97.

[16]  Wouter M. Koolen,et al.  Safe Testing , 2019, 2020 Information Theory and Applications Workshop (ITA).

[17]  V. Vovk Conformal e-prediction for change detection , 2020, 2006.02329.

[18]  Vladimir Vovk,et al.  A note on data splitting with e-values: online appendix to my comment on Glenn Shafer's "Testing by betting". , 2020, 2008.11474.

[19]  Vladimir Vovk,et al.  Game‐Theoretic Foundations for Probability and Finance , 2019, Wiley Series in Probability and Statistics.

[20]  M. Basseville,et al.  Sequential Analysis: Hypothesis Testing and Changepoint Detection , 2014 .

[21]  Mark Herbster,et al.  Tracking the Best Expert , 1995, Machine Learning.

[22]  Alexander Gammerman,et al.  Retrain or not retrain: Conformal test martingales for change-point detection , 2021, COPA.

[23]  Stephen E. Fienberg,et al.  Testing Statistical Hypotheses , 2005 .

[24]  V. Vovk,et al.  E-values: Calibration, combination, and applications , 2019 .