Simple Learning Algorithms for Decision Trees and Multivariate Polynomials

In this paper we develop a new approach for learning decision trees and multivariate polynomials via interpolation of multivariate polynomials. This new approach yields simple learning algorithms for multivariate polynomials and decision trees over finite fields under any constant bounded product distribution. The output hypothesis is a (single) multivariate polynomial that is an $\epsilon$-approximation of the target under any constant bounded product distribution. The new approach demonstrates the learnability of many classes under any constant bounded product distribution and using membership queries, such as j-disjoint disjunctive normal forms (DNFs) and multivariate polynomials with bounded degree over any field. The technique shows how to interpolate multivariate polynomials with bounded term size from membership queries only. This, in particular, gives a learning algorithm for an O(log n)-depth decision tree from membership queries only and a new learning algorithm of any multivariate polynomial over sufficiently large fields from membership queries only. We show that our results for learning from membership queries only are the best possible.

[1]  Jacob T. Schwartz,et al.  Fast Probabilistic Algorithms for Verification of Polynomial Identities , 1980, J. ACM.

[2]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[3]  Michael Ben-Or,et al.  A deterministic algorithm for sparse multivariate polynomial interpolation , 1988, STOC '88.

[4]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.

[5]  Richard Zippel,et al.  Interpolating Polynomials from Their Values , 1990, J. Symb. Comput..

[6]  Marek Karpinski,et al.  Fast Parallel Algorithms for Sparse Multivariate Polynomial Interpolation over Finite Fields , 1988, SIAM J. Comput..

[7]  Ron M. Roth,et al.  Interpolation and Approximation of Sparse Multivariate Polynomials over GF(2) , 1991, SIAM J. Comput..

[8]  Eyal Kushilevitz,et al.  Learning decision trees using the Fourier spectrum , 1991, STOC '91.

[9]  Marek Karpinski,et al.  On Zero-Testing and Interpolation of k-Sparse Multivariate Polynomials Over Finite Fields , 1991, Theor. Comput. Sci..

[10]  Noga Alon,et al.  The Probabilistic Method , 2015, Fundamentals of Ramsey Theory.

[11]  Nader H. Bshouty,et al.  Exact learning via the Monotone theory , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[12]  Tadao Kasami,et al.  A Polynomial Time Learning Algorithm for Recognizable Series , 1994 .

[13]  Francesco Bergadano,et al.  Learning Behaviors of Automata from Multiplicity and Equivalence Queries , 1994, SIAM J. Comput..

[14]  Roni Khardon On Using the Fourier Transform to Learn Disjoint DNF , 1994, Inf. Process. Lett..

[15]  Jeffrey C. Jackson,et al.  An efficient membership-query algorithm for learning DNF with respect to the uniform distribution , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[16]  Nader H. Bshouty A note on learning multivariate polynomials under the uniform distribution (extended abstract) , 1995, COLT '95.

[17]  Nader H. Bshouty Exact Learning Boolean Function via the Monotone Theory , 1995, Inf. Comput..

[18]  Yishay Mansour,et al.  Randomized Interpolation and Approximation of Sparse Polynomials , 1992, SIAM J. Comput..

[19]  Linda Sellie,et al.  Learning Sparse Multivariate Polynomials over a Field with Queries and Counterexamples , 1996, J. Comput. Syst. Sci..

[20]  Francesco Bergadano,et al.  Learning Sat-k-DNF formulas from membership queries , 1996, STOC '96.

[21]  Eyal Kushilevitz,et al.  On the applications of multiplicity automata in learning , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[22]  A. Blum Learning Boolean Functions in an Infinite Attribute Space , 1992, Machine Learning.