Learning Functions of Halfspaces using Prefix Covers

We present a simple query-algorithm for learning arbitrary functions of k halfspaces under any product distribution on the Boolean hypercube. Our algorithms learn any function of k halfspaces to within accuracy e in time O((nk/e)) under any product distribution on {0, 1} using read-once branching programs as a hypothesis.. This gives the first poly(n, 1/e) algorithm for learning even the intersection of 2 halfspaces under the uniform distribution on {0, 1}; previously known algorithms had an exponential dependence either on the accuracy parameter e or the dimension n. To prove this result, we identify a new structural property of Boolean functions that yields learnability with queries: that of having a small prefix cover.

[1]  David Zuckerman,et al.  Pseudorandom Generators for Polynomial Threshold Functions , 2013, SIAM J. Comput..

[2]  Parikshit Gopalan,et al.  Polynomial-Time Approximation Schemes for Knapsack and Related Counting Problems using Branching Programs , 2010, Electron. Colloquium Comput. Complex..

[3]  Santosh S. Vempala,et al.  A random sampling based algorithm for learning the intersection of half-spaces , 1997, Proceedings 38th Annual Symposium on Foundations of Computer Science.

[4]  ERIC B. BAUM,et al.  On learning a union of half spaces , 1990, J. Complex..

[5]  Nader H. Bshouty,et al.  On Learning width Two Branching Programs , 1998, Inf. Process. Lett..

[6]  J. C. Jackson.  Learning Functions Represented as Multiplicity Automata , 1997 .

[7]  Philip M. Long,et al.  Baum's Algorithm Learns Intersections of Halfspaces with Respect to Log-Concave Distributions , 2009, APPROX-RANDOM.

[8]  David Zuckerman,et al.  Deterministic extractors for small-space sources , 2011, J. Comput. Syst. Sci..

[9]  Santosh S. Vempala,et al.  Learning Convex Concepts from Gaussian Distributions with PCA , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[10]  Ronitt Rubinfeld,et al.  On learning bounded-width branching programs , 1995, COLT '95.

[11]  Prahladh Harsha,et al.  An invariance principle for polytopes , 2009, JACM.

[12]  Santosh S. Vempala,et al.  A random-sampling-based algorithm for learning intersections of halfspaces , 2010, JACM.

[13]  David A. Mix Barrington,et al.  Bounded-width polynomial-size branching programs recognize exactly those languages in NC1 , 1986, STOC '86.

[14]  Rocco A. Servedio,et al.  Learning intersections and thresholds of halfspaces , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[15]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[16]  Ryan O'Donnell,et al.  Learning Geometric Concepts via Gaussian Surface Area , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[17]  Avrim Blum,et al.  Learning an Intersection of a Constant Number of Halfspaces over a Uniform Distribution , 1997, J. Comput. Syst. Sci..

[18]  Rocco A. Servedio,et al.  Agnostically learning halfspaces , 2005, 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS'05).

[19]  Noam Nisan,et al.  Constant depth circuits, Fourier transform, and learnability , 1993, JACM.

[20]  Leslie G. Valiant,et al.  Cryptographic Limitations on Learning Boolean Formulae and Finite Automata , 1993, Machine Learning: From Theory to Applications.

[21]  Eric Vigoda,et al.  An FPTAS for #Knapsack and Related Counting Problems , 2011, 2011 IEEE 52nd Annual Symposium on Foundations of Computer Science.