Self-Directed Learning and Its Relation to the VC-Dimension and to Teacher-Directed Learning

We study the self-directed (SD) learning model. In this model a learner chooses examples, guesses their classification and receives immediate feedback indicating the correctness of its guesses. We consider several fundamental questions concerning this model: the parameters of a task that determine the cost of learning, the computational complexity of a student, and the relationship between this model and the teacher-directed (TD) learning model. We answer the open problem of relating the cost of self-directed learning to the VC-dimension by showing that no such relation exists. Furthermore, we refute the conjecture that for the intersection-closed case, the cost of self-directed learning is bounded by the VC-dimension. We also show that the cost of SD learning may be arbitrarily higher that that of TD learning.Finally, we discuss the number of queries needed for learning in this model and its relationship to the number of mistakes the student incurs. We prove a trade-off formula showing that an algorithm that makes fewer queries throughout its learning process, necessarily suffers a higher number of mistakes.

[1]  Wolfgang Maass,et al.  Lower Bound Methods and Separation Results for On-Line Learning Models , 2004, Machine Learning.

[2]  Avrim Blum,et al.  Separating distribution-free and mistake-bound learning models over the Boolean domain , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[3]  Wolfgang Maass,et al.  On-line learning with an oblivious environment and the power of randomization , 1991, COLT '91.

[4]  N. Littlestone.  Mistake bounds and logarithmic linear-threshold learning algorithms , 1990 .

[5]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[6]  Shai Ben-David,et al.  Online learning versus offline learning , 1995, EuroCOLT.

[7]  Ronald L. Rivest,et al.  Learning Binary Relations and Total Orders , 1989, COLT 1989.

[8]  Yiqun Lisa Yin.  Reducing the number of queries in self-directed learning , 1995, COLT '95.

[9]  Mihalis Yannakakis,et al.  On limited nondeterminism and the complexity of the V-C dimension , 1993, [1993] Proceedings of the Eigth Annual Structure in Complexity Theory Conference.

[10]  Avrim Blum.  Rank-r Decision Trees are a Subclass of r-Decision Lists , 1992, Inf. Process. Lett..

[11]  Ronald L. Rivest,et al.  Learning binary relations, total orders, and read-once formulas , 1990 .

[12]  Avrim Blum.  Learning boolean functions in an infinite attribute space , 1990, STOC '90.

[13]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.

[14]  Shai Ben-David,et al.  On self-directed learning , 1995, COLT '95.

[15]  Sally A. Goldman,et al.  The Power of Self-Directed Learning , 1994, Machine Learning.

[16]  Manfred K. Warmuth,et al.  The weighted majority algorithm , 1989, 30th Annual Symposium on Foundations of Computer Science.

[17]  Zhixiang Chen,et al.  On-line learning of rectangles and unions of rectangles , 2004, Machine Learning.

[18]  David Haussler,et al.  Learning decision trees from random examples , 1988, COLT '88.

[19]  Ami Litman,et al.  Optimal Mistake Bound Learning is Hard , 1998, Inf. Comput..

[20]  Philip M. Long,et al.  Apple tasting and nearly one-sided learning , 1992, Proceedings., 33rd Annual Symposium on Foundations of Computer Science.

[21]  Michael Kearns,et al.  On the complexity of teaching , 1991, COLT '91.

[22]  Ronald L. Rivest,et al.  Being taught can be faster than asking questions , 1995, COLT '95.