Inapproximability of VC Dimension and Littlestone's Dimension

We study the complexity of computing the VC Dimension and Littlestone's Dimension. Given an explicit description of a finite universe and a concept class (a binary matrix whose $(x,C)$-th entry is $1$ iff element $x$ belongs to concept $C$), both can be computed exactly in quasi-polynomial time ($n^{O(\log n)}$). Assuming the randomized Exponential Time Hypothesis (ETH), we prove nearly matching lower bounds on the running time, that hold even for approximation algorithms.

[1]  Hans Ulrich Simon,et al.  Recursive teaching dimension, VC-dimension and sample compression , 2014, J. Mach. Learn. Res..

[2]  KearnsMichael,et al.  Cryptographic limitations on learning Boolean formulae and finite automata , 1994 .

[3]  Alexander A. Sherstov,et al.  Cryptographic Hardness for Learning Intersections of Halfspaces , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[4]  Mark Braverman,et al.  Approximating the best Nash Equilibrium in no(log n)-time breaks the Exponential Time Hypothesis , 2015, Electron. Colloquium Comput. Complex..

[5]  Shay Moran,et al.  Compressing and Teaching for Low VC-Dimension , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[6]  Pasin Manurangsi,et al.  Almost-polynomial ratio ETH-hardness of approximating densest k-subgraph , 2016, STOC.

[7]  Sanjeev Arora,et al.  Finding overlapping communities in social networks: toward a rigorous approach , 2011, EC '12.

[8]  Vitaly Feldman,et al.  New Results for Learning Noisy Parities and Halfspaces , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[9]  Prasad Raghavendra,et al.  A Birthday Repetition Theorem and Complexity of Approximating Dense CSPs , 2016, ICALP.

[10]  Rocco A. Servedio,et al.  Agnostically learning halfspaces , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[11]  Avrim Blum,et al.  Separating distribution-free and mistake-bound learning models over the Boolean domain , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[12]  Elchanan Mossel,et al.  On the complexity of approximating the VC dimension , 2001, Proceedings 16th Annual IEEE Conference on Computational Complexity.

[13]  Marcus Schaefer,et al.  Deciding the k-dimension is PSPACE-complete , 2000, Proceedings 15th Annual IEEE Conference on Computational Complexity.

[14]  Aranyak Mehta,et al.  Playing large games using simple strategies , 2003, EC '03.

[15]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[16]  Yu Cheng,et al.  Hardness Results for Signaling in Bayesian Zero-Sum and Network Routing Games , 2015, EC.

[17]  Adam R. Klivans.  Cryptographic Hardness of Learning , 2016, Encyclopedia of Algorithms.

[18]  Ami Litman,et al.  Optimal Mistake Bound Learning is Hard , 1998, Inf. Comput..

[19]  Aviad Rubinstein,et al.  Detecting communities is Hard (And Counting Them is Even Harder) , 2016, ITCS.

[20]  Siddharth Barman,et al.  Approximating Nash Equilibria and Dense Bipartite Subgraphs via an Approximate Version of Caratheodory's Theorem , 2015, STOC.

[21]  John Fearnley,et al.  Inapproximability Results for Approximate Nash Equilibria , 2016, WINE.

[22]  Russell Impagliazzo,et al.  AM with Multiple Merlins , 2014, 2014 IEEE 29th Conference on Computational Complexity (CCC).

[23]  Nathan Linial,et al.  Results on learnability and the Vapnik-Chervonenkis dimension , 1988, [Proceedings 1988] 29th Annual Symposium on Foundations of Computer Science.

[24]  Amit Daniely,et al.  Complexity Theoretic Limitations on Learning DNF's , 2014, COLT.

[25]  Aviad Rubinstein,et al.  Settling the Complexity of Computing Approximate Two-Player Nash Equilibria , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[26]  Amit Daniely,et al.  Complexity theoretic limitations on learning halfspaces , 2015, STOC.

[27]  Shai Ben-David,et al.  Self-Directed Learning and Its Relation to the VC-Dimension and to Teacher-Directed Learning , 2004, Machine Learning.

[28]  Ran Raz,et al.  Two Query PCP with Sub-Constant Error , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[29]  Russell Impagliazzo,et al.  Complexity of k-SAT , 1999, Proceedings. Fourteenth Annual IEEE Conference on Computational Complexity (Formerly: Structure in Complexity Theory Conference) (Cat.No.99CB36317).

[30]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[31]  Mihalis Yannakakis,et al.  On limited nondeterminism and the complexity of the V-C dimension , 1993, [1993] Proceedings of the Eigth Annual Structure in Complexity Theory Conference.

[32]  Cristina Bazgan,et al.  On the Approximability of Partial VC Dimension , 2016, COCOA.

[33]  Michael Kharitonov,et al.  Cryptographic lower bounds for learnability of Boolean functions on the uniform distribution , 1992, COLT '92.

[34]  Russell Impagliazzo,et al.  Which problems have strongly exponential complexity? , 1998, Proceedings 39th Annual Symposium on Foundations of Computer Science (Cat. No.98CB36280).

[35]  Leslie G. Valiant,et al.  Cryptographic Limitations on Learning Boolean Formulae and Finite Automata , 1993, Machine Learning: From Theory to Applications.

[36]  M. Schaefer.  Deciding the VapnikCervonenkis Dimension is ? p 3 -complete , 1999 .

[37]  Michael Kharitonov,et al.  Cryptographic hardness of distribution-specific learning , 1993, STOC.

[38]  Li Han,et al.  Mixture Selection, Mechanism Design, and Signaling , 2015, 2015 IEEE 56th Annual Symposium on Foundations of Computer Science.

[39]  Aviad Rubinstein,et al.  ETH-Hardness for Signaling in Symmetric Zero-Sum Games , 2015, ArXiv.

[40]  Omri Weinstein,et al.  ETH Hardness for Densest-k-Subgraph with Perfect Completeness , 2015, SODA.

[41]  Yakov Babichenko,et al.  Can Almost Everybody be Almost Happy? , 2015, ITCS.

[42]  Shai Ben-David,et al.  Agnostic Online Learning , 2009, COLT.