PAC Learning under Helpful Distributions

A PAC model under helpful distributions is introduced. A teacher associates a teaching set with each target concept and we only consider distributions such that each example in the teaching set has a non-zero weight. The performance of a learning algorithm depends on the probabilities of the examples in this teaching set. In this model, an Occam's razor theorem and its converse are proved. The class of decision lists is proved PAC learnable under helpful distributions. A PAC learning model with simple teacher (simplicity is based on program-size complexity) is also defined and the model is compared with other models of teaching.

[1]  M. Kearns,et al.  On the complexity of teaching , 1991, COLT '91.

[2]  David Haussler,et al.  Equivalence of models for polynomial learnability , 1988, COLT '88.

[3]  Nader H. Bshouty.  Exact Learning Boolean Function via the Monotone Theory , 1995, Inf. Comput..

[4]  M. Kearns,et al.  Recent Results on Boolean Concept Learning , 1987 .

[5]  Alon Itai,et al.  Nonuniform Learnability , 1988, J. Comput. Syst. Sci..

[6]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[7]  Sally A. Goldman,et al.  Teaching a Smarter Learner , 1996, J. Comput. Syst. Sci..

[8]  Leonard Pitt,et al.  Inductive Inference, DFAs, and Computational Complexity , 1989, AII.

[9]  H. David Mathias,et al.  DNF—if you can't learn'em, teach'em: an interactive model of teaching , 1995, COLT '95.

[10]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[11]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[12]  B. Natarajan.  Machine Learning: A Theoretical Approach , 1992 .

[13]  Colin de la Higuera.  Characteristic Sets for Polynomial Grammatical Inference , 1997 .

[14]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1993, Graduate Texts in Computer Science.

[15]  Leonard Pitt,et al.  On the Necessity of Occam Algorithms , 1992, Theor. Comput. Sci..

[16]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[17]  Paul M. B. Vitányi,et al.  A Theory of Learning Simple Concepts Under Simple Distributions , 1989, COLT 1989.

[18]  Rémi Gilleron,et al.  PAC Learning with Simple Examples , 1996, STACS.

[19]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[20]  Kathleen Romanik,et al.  Approximate testing and learnability , 1992, COLT '92.

[21]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[22]  J. Oncina,et al.  INFERRING REGULAR LANGUAGES IN POLYNOMIAL UPDATED TIME , 1992 .

[23]  José L. Balcázar,et al.  Simple PAC Learning of Simple Decision Lists , 1995, ALT.

[24]  Leslie G. Valiant,et al.  On the learnability of Boolean formulae , 1987, STOC.

[25]  Rajesh Parekh,et al.  Simple DFA are Polynomially Probably Exactly Learnable from Simple Examples , 1999, ICML.

[26]  Tao Jiang,et al.  Lower Bounds on Learning Decision Lists and Trees , 1995, Inf. Comput..

[27]  Balas K. Natarajan,et al.  On learning Boolean functions , 1987, STOC.

[28]  Simon Kasif,et al.  Learning with a Helpful Teacher , 1991, IJCAI.

[29]  E. Mark Gold,et al.  Complexity of Automaton Identification from Given Data , 1978, Inf. Control..

[30]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[31]  Alon Itai,et al.  Learnability by fixed distributions , 1988, COLT '88.