On learning visual concepts and DNF formulae

We consider the problem of learning DNF formulae in the mistake-bound and the PAC models. We develop a new approach, which is called polynomial explainability, that is shown to be useful for learning some new subclasses of DNF (and CNF) formulae that were not known to be learnable before. Unlike previous learnability results for DNF (and CNF) formulae, these subclasses are not limited in the number of terms or in the number of variables per term; yet, they contain the subclasses of κ-DNF and κ-term-DNF (and the corresponding classes of CNF) as special cases. We apply our DNF results to the problem of learning visual concepts and obtain learning algorithms for several natural subclasses of visual concepts that appear to have no natural boolean counterpart. On the other hand, we show that learning some other natural subclasses of visual concepts is as hard as learning the class of all DNF formulae. We also consider the robustness of these results under various types of noise.

[1]  Leonard Pitt,et al.  A polynomial-time algorithm for learning k-variable pattern languages from examples , 1989, COLT '89.

[2]  Ming Li,et al.  Learning in the Presence of Malicious Errors , 1993, SIAM J. Comput..

[3]  Vijay Raghavan,et al.  Read-Twice DNF Formulas are Properly Learnable , 1994, Inf. Comput..

[4]  Scott E. Decatur.  Statistical queries and faulty PAC oracles , 1993, COLT '93.

[5]  Eyal Kushilevitz,et al.  On learning Read-k-Satisfy-j DNF , 1994, COLT '94.

[6]  Nader H. Bshouty,et al.  Exact learning via the Monotone theory , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[7]  Thomas R. Hancock,et al.  Learning 2u DNF formulas and ku decision trees , 1991, COLT 1991.

[8]  Leslie G. Valiant,et al.  On the learnability of Boolean formulae , 1987, STOC.

[9]  Eyal Kushilevitz,et al.  Learning Decision Trees Using the Fourier Spectrum , 1993, SIAM J. Comput..

[10]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[11]  Thomas R. Hancock,et al.  Learning 2µ DNF Formulas and kµ Decision Trees , 1991, COLT.

[12]  Ming Li,et al.  A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution , 1989, 30th Annual Symposium on Foundations of Computer Science.

[13]  H. Shvaytser,et al.  Learnable and nonlearnable visual concepts , 1990 .

[14]  N. Littlestone.  Mistake bounds and logarithmic linear-threshold learning algorithms , 1990 .

[15]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[16]  Robert E. Schapire,et al.  Pattern languages are not learnable , 1990, Annual Conference Computational Learning Theory.

[17]  Avrim Blum,et al.  Fast learning of k-term DNF formulas with queries , 1992, STOC '92.

[18]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[19]  Leslie G. Valiant,et al.  Learning Disjunction of Conjunctions , 1985, IJCAI.

[20]  Mark Jerrum.  Simple Translation-Invariant Concepts Are Hard to Learn , 1994, Inf. Comput..

[21]  Leonard Pitt,et al.  Exact learning of read-k disjoint DNF and not-so-disjoint DNF , 1992, COLT '92.

[22]  David Haussler,et al.  Equivalence of models for polynomial learnability , 1988, COLT '88.

[23]  Javed A. Aslam,et al.  General bounds on statistical query learning and PAC learning with noise via hypothesis boosting , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[24]  Dana Angluin,et al.  Computational learning theory: survey and selected bibliography , 1992, STOC '92.

[25]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[26]  H. Aizenstein,et al.  Exact learning of read-twice DNF formulas , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[27]  Dana Angluin,et al.  Finding Patterns Common to a Set of Strings , 1980, J. Comput. Syst. Sci..

[28]  George Shackelford,et al.  Learning k-DNF with noise in the attributes , 1988, Annual Conference Computational Learning Theory.

[29]  Lisa Hellerstein,et al.  Read-thrice DNF is hard to learn with membership and equivalence queries , 1992, Proceedings., 33rd Annual Symposium on Foundations of Computer Science.