An Efficient Membership-Query Algorithm for Learning DNF with Respect to the Uniform Distribution

We present a membership-query algorithm for efficiently learning DNF with respect to the uniform distribution. In fact, the algorithm properly learns with respect to uniform the class TOP of Boolean functions expressed as a majority vote over parity functions. We also describe extensions of this algorithm for learning DNF over certain nonuniform distributions and for learning a class of geometric concepts that generalizes DNF. Furthermore, we show that DNF is weakly learnable with respect to uniform from noisy examples. Our strong learning algorithm utilizes one of Freund's boosting techniques and relies on the fact that boosting does not require a completely distribution-independent weak learner. The boosted weak learner is a nonuniform extension of a parity-finding algorithm discovered by Goldreich and Levin.

[1]  Leslie G. Valiant,et al.  On the learnability of Boolean formulae , 1987, STOC.

[2]  Eyal Kushilevitz,et al.  Learning Decision Trees Using the Fourier Spectrum , 1993, SIAM J. Comput..

[3]  Lisa Hellerstein,et al.  Read-thrice DNF is hard to learn with membership and equivalence queries , 1992, Proceedings., 33rd Annual Symposium on Foundations of Computer Science.

[4]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[5]  Eyal Kushilevitz,et al.  On learning visual concepts and DNF formulae , 1993, COLT '93.

[6]  H. Aizenstein,et al.  Exact learning of read-twice DNF formulas , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[7]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1997, EuroCOLT.

[8]  Paul W. Goldberg,et al.  Learning unions of boxes with membership and equivalence queries , 1994, COLT '94.

[9]  Noam Nisan,et al.  Constant depth circuits, Fourier transform, and learnability , 1993, JACM.

[10]  Richard J. Lipton,et al.  Amplification of weak learning under the uniform distribution , 1993, COLT '93.

[11]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[12]  Yoav Freund,et al.  An improved boosting algorithm and its implications on learning complexity , 1992, COLT '92.

[13]  Mark Craven,et al.  Learning Sparse Perceptrons , 1995, NIPS.

[14]  Philip M. Long,et al.  Composite geometric concepts and polynomial predictability , 1990, COLT '90.

[15]  Nader H. Bshouty,et al.  Exact learning via the Monotone theory , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[16]  Michael Frazier,et al.  Learning conjunctions of Horn clauses , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[17]  Michael Kharitonov,et al.  Cryptographic hardness of distribution-specific learning , 1993, STOC.

[18]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[19]  J. C. Jackson.  The harmonic sieve: a novel application of Fourier analysis to machine learning theory and practice , 1996 .

[20]  Jehoshua Bruck,et al.  Harmonic Analysis of Polynomial Threshold Functions , 1990, SIAM J. Discret. Math..

[21]  Zhixiang Chen,et al.  On learning discretized geometric concepts , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[22]  Jeffrey C. Jackson,et al.  An efficient membership-query algorithm for learning DNF with respect to the uniform distribution , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[23]  Alfred V. Aho,et al.  The Design and Analysis of Computer Algorithms , 1974 .

[24]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1990, COLT '90.

[25]  D. Angluin.  Queries and Concept Learning , 1988 .

[26]  Yishay Mansour,et al.  Weakly learning DNF and characterizing statistical query learning using Fourier analysis , 1994, STOC '94.

[27]  Robert E. Schapire,et al.  Exact Identification of Read-Once Formulas Using Fixed Points of Amplification Functions , 1993, SIAM J. Comput..

[28]  Dana Angluin,et al.  When won't membership queries help? , 1991, STOC '91.

[29]  Pavel Pudlák,et al.  On the computational power of depth 2 circuits with threshold and modulo gates , 1994, STOC '94.

[30]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[31]  Leonard Pitt,et al.  Exact learning of read-k disjoint DNF and not-so-disjoint DNF , 1992, COLT '92.

[32]  Yishay Mansour,et al.  An O(nlog log n) learning algorithm for DNF under the uniform distribution , 1992, COLT '92.

[33]  Leonid A. Levin,et al.  A hard-core predicate for all one-way functions , 1989, STOC '89.