On learning visual concepts and DNF formulae

We consider the problem of learning DNF formulae in the mistake-bound and the PAC models. We develop a new approach, which is called polynomial explainability, that is shown to be useful for learning some new subclasses of DNF (and CNF) formulae that were not known to be learnable before. Unlike previous learnability results for DNF (and CNF) formulae, these subclasses are not limited in the number of terms or in the number of variables per term; yet, they contain the subclasses of k-DNF and k-term-DNF (and the corresponding classes of CNF) as special cases. We apply our DNF results to the problem of learning visual concepts and obtain learning algorithms for several natural subclasses of visual concepts that appear to have no natural boolean counterpart. On the other hand, we show that learning some other natural subclasses of visual concepts is as hard as learning the class of all DNF formulae. We also consider the robustness of these results under various types of noise.

[1]  Leslie G. Valiant,et al.  On the learnability of Boolean formulae , 1987, STOC.

[2]  Tricia Walker,et al.  Computer science , 1996, English for academic purposes series.

[3]  Scott E. Decatur.  Statistical queries and faulty PAC oracles , 1993, COLT '93.

[4]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[5]  Michael Frazier,et al.  Learning Conjunctions of Horn Clauses**An extended abstract of this paper appears in Proceedings of the 31st Annual Symposium on Foundations of Computer Science, IEEE Computer Society Press, 1990. , 1990 .

[6]  Haim Schweitzer.  Learnable and Nonlearnable Visual Concepts , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Jeffrey C. Jackson,et al.  An efficient membership-query algorithm for learning DNF with respect to the uniform distribution , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[8]  Nader H. Bshouty,et al.  Exact learning via the Monotone theory , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[9]  N. S. Barnett,et al.  Private communication , 1969 .

[10]  Avrim Blum.  Learning boolean functions in an infinite attribute space , 1990, STOC '90.

[11]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[12]  Leonard Pitt,et al.  Exact learning of read-k disjoint DNF and not-so-disjoint DNF , 1992, COLT '92.

[13]  Javed A. Aslam,et al.  General bounds on statistical query learning and PAC learning with noise via hypothesis boosting , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[14]  Dana Angluin,et al.  Computational learning theory: survey and selected bibliography , 1992, STOC '92.

[15]  Robert E. Schapire,et al.  Pattern languages are not learnable , 1990, Annual Conference Computational Learning Theory.

[16]  Temple F. Smith.  Occam's razor , 1980, Nature.

[17]  George Shackelford,et al.  Learning k-DNF with noise in the attributes , 1988, Annual Conference Computational Learning Theory.

[18]  David Haussler,et al.  Equivalence of models for polynomial learnability , 1988, COLT '88.

[19]  Mark Jerrum.  Simple Translation-Invariant Concepts Are Hard to Learn , 1994, Inf. Comput..

[20]  Eyal Kushilevitz,et al.  Learning decision trees using the Fourier spectrum , 1991, STOC '91.

[21]  Leonard Pitt,et al.  A polynomial-time algorithm for learning k-variable pattern languages from examples , 1989, COLT '89.

[22]  Thomas R. Hancock,et al.  Learning 2µ DNF Formulas and kµ Decision Trees , 1991, COLT.

[23]  Ming Li,et al.  A theory of learning simple concepts under simple distributions and average case complexity for the universal distribution , 1989, 30th Annual Symposium on Foundations of Computer Science.

[24]  N. Littlestone.  Mistake bounds and logarithmic linear-threshold learning algorithms , 1990 .

[25]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[26]  H. Aizenstein,et al.  Exact learning of read-twice DNF formulas , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[27]  Vijay Raghavan,et al.  Read-Twice DNF Formulas are Properly Learnable , 1994, Inf. Comput..

[28]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[29]  Avrim Blum,et al.  Fast learning of k-term DNF formulas with queries , 1992, STOC '92.

[30]  Dana Angluin,et al.  Finding Patterns Common to a Set of Strings , 1980, J. Comput. Syst. Sci..

[31]  Eyal Kushilevitz,et al.  On learning Read-k-Satisfy-j DNF , 1994, COLT '94.