Labeled Compression Schemes for Extremal Classes

It is a long-standing open problem whether there always exists a compression scheme whose size is of the order of the Vapnik-Chervonienkis (VC) dimension $d$. Recently compression schemes of size exponential in $d$ have been found for any concept class of VC dimension $d$. Previously, compression schemes of size $d$ have been given for maximum classes, which are special concept classes whose size equals an upper bound due to Sauer-Shelah. We consider a generalization of maximum classes called extremal classes. Their definition is based on a powerful generalization of the Sauer-Shelah bound called the Sandwich Theorem, which has been studied in several areas of combinatorics and computer science. The key result of the paper is a construction of a sample compression scheme for extremal classes of size equal to their VC dimension. We also give a number of open problems concerning the combinatorial structure of extremal classes and the existence of unlabeled compression schemes for them.

[1]  Shay Moran,et al.  Teaching and compressing for low VC-dimension , 2015, Electron. Colloquium Comput. Complex..

[2]  David Newnham.  Shattering news. , 2016, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[3]  Lajos Rónyai,et al.  Shattering-Extremal Set Systems of VC Dimension at most 2 , 2014, Electron. J. Comb..

[4]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[5]  J. Lawrence.  Lopsided sets and orthant-intersection by convex sets , 1983 .

[6]  Benjamin I. P. Rubinstein,et al.  A Geometric Approach to Sample Compression , 2009, J. Mach. Learn. Res..

[7]  Manfred K. Warmuth,et al.  Sample compression, learnability, and the Vapnik-Chervonenkis dimension , 1995, Machine Learning.

[8]  Shay Moran,et al.  Shattering-Extremal Systems , 2012, ArXiv.

[9]  Manfred K. Warmuth,et al.  Unlabeled Compression Schemes for Maximum Classes, , 2007, COLT.

[10]  Peter L. Bartlett,et al.  Shifting: One-inclusion mistake bounds and sample compression , 2009, J. Comput. Syst. Sci..

[11]  Hans Ulrich Simon,et al.  Recursive Teaching Dimension, Learning Complexity, and Maximum Classes , 2010, ALT.

[12]  Temple F. Smith.  Occam's razor , 1980, Nature.

[13]  Shay Moran,et al.  Sample compression schemes for VC classes , 2015, 2016 Information Theory and Applications Workshop (ITA).

[14]  Yoav Freund,et al.  Boosting: Foundations and Algorithms , 2012 .

[15]  S. Shelah.  A combinatorial problem; stability and order for models and theories in infinitary languages. , 1972 .

[16]  Shai Ben-David,et al.  Combinatorial Variability of Vapnik-chervonenkis Classes with Applications to Sample Compression Schemes , 1998, Discret. Appl. Math..

[17]  Andreas W. M. Dress,et al.  Towards a theory of holistic clustering , 1996, Mathematical Hierarchies and Biology.

[18]  Shay Moran,et al.  Shattering, Graph Orientations, and Connectivity , 2012, Electron. J. Comb..

[19]  Roi Livni,et al.  Honest Compressions and Their Application to Compression Schemes , 2013, COLT.

[20]  Norbert Sauer,et al.  On the Density of Families of Sets , 1972, J. Comb. Theory A.

[21]  Béla Bollobás,et al.  Reverse Kleitman Inequalities , 1989 .

[22]  David Haussler,et al.  Predicting {0,1}-functions on randomly drawn points , 1988, COLT '88.

[23]  Manfred K. Warmuth.  Compressing to VC Dimension Many Points , 2003, COLT.

[24]  Manfred K. Warmuth,et al.  Relating Data Compression and Learnability , 2003 .

[25]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[26]  Pierre Simon,et al.  Externally definable sets and dependent pairs II , 2012, 1202.2650.

[27]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[28]  Amit Daniely,et al.  Optimal learners for multiclass problems , 2014, COLT.

[29]  R Lajos.  Shattering-extremal set systems of VC dimension at most 2 , 2014 .

[30]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[31]  Manfred K. Warmuth,et al.  Learning integer lattices , 1990, COLT '90.

[32]  Emo Welzl,et al.  Vapnik-Chervonenkis dimension and (pseudo-)hyperplane arrangements , 1994, Discret. Comput. Geom..

[33]  Sally Floyd,et al.  Space-bounded learning and the Vapnik-Chervonenkis dimension , 1989, COLT '89.

[34]  Boting Yang,et al.  Generalizing Labeled and Unlabeled Sample Compression to Multi-label Concept Classes , 2014, ALT.

[35]  B. Maurey,et al.  Sous-espaces $l^P$ des espaces de Banach , 1983 .

[36]  Shay Moran,et al.  Compressing and Teaching for Low VC-Dimension , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[37]  Peter L. Bartlett,et al.  Bounding Embeddings of VC Classes into Maximum Classes , 2014, ArXiv.

[38]  Lajos Rónyai,et al.  Some Combinatorial Applications of Gröbner Bases , 2011, CAI.

[39]  Alex M. Andrew,et al.  Boosting: Foundations and Algorithms , 2012 .

[40]  G. Greco,et al.  Embeddings and the Trace of Finite Sets , 1998, Inf. Process. Lett..

[41]  Hans-Jürgen Bandelt,et al.  Combinatorics of lopsided sets , 2006, Eur. J. Comb..

[42]  Béla Bollobás,et al.  Defect Sauer Results , 1995, J. Comb. Theory A.