A formal proof of PAC learnability for decision stumps

We present a formal proof in Lean of probably approximately correct (PAC) learnability of the concept class of decision stumps. This classic result in machine learning theory derives a bound on error probabilities for a simple type of classifier. Though such a proof appears simple on paper, analytic and measure-theoretic subtleties arise when carrying it out fully formally. Our proof is structured so as to separate reasoning about deterministic properties of a learning function from proofs of measurability and analysis of probabilities.

[1]  Nikolaj Bjørner,et al.  Guiding High-Performance SAT Solvers with Unsat-Core Predictions , 2019, SAT.

[2]  Roman Fric,et al.  A Categorical Approach to Probability Theory , 2010, Stud Logica.

[3]  Bas Spitters,et al.  Synthetic topology in Homotopy Type Theory for probabilistic programming , 2019, ArXiv.

[4]  Bruno Blanchet,et al.  A Computationally Sound Mechanized Prover for Security Protocols , 2008, IEEE Transactions on Dependable and Secure Computing.

[5]  Ilya Sergey,et al.  Certifying Certainty and Uncertainty in Approximate Membership Query Structures , 2020, CAV.

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory, Second Edition , 2000, Statistics for Engineering and Information Science.

[7]  Cezary Kaliszyk,et al.  HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving , 2017, ICLR.

[8]  Tobias Nipkow,et al.  A FORMAL PROOF OF THE KEPLER CONJECTURE , 2015, Forum of Mathematics, Pi.

[9]  Benjamin Grégoire,et al.  EasyCrypt: A Tutorial , 2013, FOSAD.

[10]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[11]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[12]  Nadav Cohen,et al.  On the Expressive Power of Deep Learning: A Tensor Analysis , 2015, COLT 2016.

[13]  Manabu Hagiwara,et al.  Formalization of Shannon’s Theorems , 2014, Journal of Automated Reasoning.

[14]  Christine Paulin-Mohring,et al.  Proofs of randomized algorithms in Coq , 2006, Sci. Comput. Program..

[15]  Joseph Tassarotti,et al.  Verified Tail Bounds for Randomized Programs , 2018, ITP.

[16]  Cezary Kaliszyk,et al.  Reinforcement Learning of Theorem Proving , 2018, NeurIPS.

[17]  Ronald L. Rivest,et al.  Cryptography and Machine Learning , 1991, ASIACRYPT.

[18]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[19]  Johannes Hölzl,et al.  A Formally Verified Proof of the Central Limit Theorem , 2014, Journal of Automated Reasoning.

[20]  Joe Hurd,et al.  Formal verification of probabilistic algorithms , 2003 .

[21]  Dawn Xiaodong Song,et al.  GamePad: A Learning Environment for Theorem Proving , 2018, ICLR.

[22]  Sarah M. Loos,et al.  HOList: An Environment for Machine Learning of Higher Order Logic Theorem Proving , 2019, ICML.

[23]  Gordon Stewart,et al.  Certifying the True Error: Machine Learning in Coq with Verified Generalization Guarantees , 2019, AAAI.

[24]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[25]  Josef Urban,et al.  Hammering Mizar by Learning Clause Guidance (Short Paper) , 2019, ITP.

[26]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[27]  Georges Gonthier,et al.  Formal Proof—The Four- Color Theorem , 2008 .

[28]  T. Hales.  Formal Proof , 2008 .

[29]  David L. Dill,et al.  Developing Bug-Free Machine Learning Systems With Formal Mathematics , 2017, ICML.

[30]  Dietrich Klakow,et al.  A Formal Proof of the Expressiveness of Deep Learning , 2017, ITP.

[31]  Jeremy Avigad,et al.  A Machine-Checked Proof of the Odd Order Theorem , 2013, ITP.

[32]  Benjamin Grégoire,et al.  Formal certification of code-based cryptographic proofs , 2009, POPL '09.

[33]  James McKinna,et al.  A Machine-Checked Proof of the Average-Case Complexity of Quicksort in Coq , 2008, TYPES.

[34]  Xavier Leroy,et al.  Formal verification of a realistic compiler , 2009, CACM.

[35]  Johannes Hölzl,et al.  Three Chapters of Measure Theory in Isabelle/HOL , 2011, ITP.

[36]  Jeremy Avigad,et al.  The Lean Theorem Prover (System Description) , 2015, CADE.

[37]  Tobias Nipkow,et al.  Verified Analysis of Random Trees , 2018 .

[38]  R. Aumann.  Borel structures for function spaces , 1961 .

[39]  Simon Peyton Jones,et al.  Type classes: an exploration of the design space , 1997 .

[40]  J. Gregory Morrisett,et al.  The Foundational Cryptography Framework , 2014, POST.

[41]  E. Berger.  UNIFORM CENTRAL LIMIT THEOREMS (Cambridge Studies in Advanced Mathematics 63) By R. M. D UDLEY : 436pp., £55.00, ISBN 0-521-46102-2 (Cambridge University Press, 1999). , 2001 .

[42]  Johannes Hölzl,et al.  Construction and stochastic applications of measure spaces in higher-order logic , 2013 .