On metric entropy, Vapnik-Chervonenkis dimension, and learnability for a class of distributions

In [23], Valiant proposed a formal framework for distribution-free concept learning which has generated a great deal of interest. A fundamental result regarding this framework was proved by Blumer et al. [6] characterizing those concept classes which are learnable in terms of their Vapnik-Chervonenkis (VC) dimension. More recently, Benedek and Itai [4] studied learnability with respect to a fixed probability distribution (a variant of the original distribution-free framework) and proved an analogous result characterizing learnability in this case. They also stated a conjecture regarding learnability for a class of distributions. In this report, we first point out that the condition for learnability obtained in [4] is equivalent to the notion of finite metric entropy (which has been studied in other contexts). Some relationships, in addition to those shown in [4], between the VC dimension of a concept class and its metric entropy with respect to various distributions are then discussed. Finally, we prove some partial results regarding learnability for a class of distributions.

[1]  V. Tikhomirov.  KOLMOGOROV'S WORK ON $ \varepsilon$-ENTROPY OF FUNCTIONAL CLASSES AND THE SUPERPOSITION OF FUNCTIONS , 1963 .

[2]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[3]  R. Dudley.  Metric Entropy of Some Classes of Sets with Differentiable Boundaries , 1974 .

[4]  John Hawkes,et al.  Hausdorff Measure, Entropy, and the Independence of Small Sets , 1974 .

[5]  R. Dudley.  Central Limit Theorems for Empirical Measures , 1978 .

[6]  Richard M. Dudley,et al.  Some special vapnik-chervonenkis classes , 1981, Discret. Math..

[7]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[8]  Leslie G. Valiant,et al.  Learning Disjunction of Conjunctions , 1985, IJCAI.

[9]  David Haussler,et al.  Classifying learnable geometric concepts with the Vapnik-Chervonenkis dimension , 1986, STOC '86.

[10]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[11]  Leslie G. Valiant,et al.  On the learnability of Boolean formulae , 1987, STOC.

[12]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[13]  David Haussler,et al.  ɛ-nets and simplex range queries , 1987, Discret. Comput. Geom..

[14]  Ming Li,et al.  Learning in the presence of malicious errors , 1993, STOC '88.

[15]  Nathan Linial,et al.  Results on learnability and the Vapnik-Chervonenkis dimension , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[16]  Prasad Tadepalli,et al.  Two New Frameworks for Learning , 1988, ML.

[17]  Jonathan Amsterdam.  Extending the Valiant Learning Model , 1988, ML.

[18]  Alon Itai,et al.  Learnability by fixed distributions , 1988, COLT '88.

[19]  Nathan Linial,et al.  Results on learnability and the Vapnick-Chervonenkis dimension , 1988, COLT '88.

[20]  David Haussler,et al.  Equivalence of models for polynomial learnability , 1988, COLT '88.

[21]  E. Baum.  Complete representations for learning from examples , 1988 .

[22]  Alon Itai,et al.  Nonuniform Learnability , 1988, J. Comput. Syst. Sci..