Noise-tolerant parallel learning of geometric concepts

We present several efficient parallel algorithms for PAC-learning geometric concepts in a constant-dimensional space. The algorithms are robust even against malicious classification noise of any rate less than 1/2. We first give an efficient noise-tolerant parallel algorithm to PAC-learn the class of geometric concepts defined by a polynomial number of (d?1)-dimensional hyperplanes against an arbitrary distribution where each hyperplane has a slope from a set of known slopes. We then describe how boosting techniques can be used so that our algorithms' dependence on?and?does not depend ond. Next, we give an efficient noise-tolerant parallel algorithm to PAC-learn the class of geometric concepts defined by a polynomial number of (d?1)-dimensional hyperplanes (of unrestricted slopes) against a uniform distribution. We then show how to extend our algorithm to learn this class against any (unknown) product distribution. Next we define a complexity measure of any setSof (d?1)-dimensional surfaces that we call thevariantofSand prove that the class of geometric concepts defined by surfaces of polynomial variant can be efficiently learned in parallel under a product distribution (even under malicious classification noise). Furthermore, we show that the VC-dimension of the class of geometric concepts defined by a single surface of variant one is ∞. Finally, we give an efficient, parallel, noise-tolerant algorithm to PAC-learn any 2-dimensional geometric concept defined by a setSof 1-dimensional surfaces of polynomial length under a uniform distribution.

[1]  Peter Auer,et al.  On-line learning of rectangles in noisy environments , 1993, COLT '93.

[2]  D. Angluin.  Queries and Concept Learning , 1988 .

[3]  David Haussler,et al.  Generalizing the PAC model: sample size bounds from metric dimension-based uniform convergence results , 1989, 30th Annual Symposium on Foundations of Computer Science.

[4]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[5]  Paul W. Goldberg,et al.  Learning unions of boxes with membership and equivalence queries , 1994, COLT '94.

[6]  Steven Homer,et al.  Learning Unions of Rectangles with Queries , 1993 .

[7]  Javed A. Aslam,et al.  Specification and simulation of statistical query algorithms for efficiency and noise tolerance , 1995, COLT '95.

[8]  D. Haussler.  Generalizing the PAC model: sample size bounds from metric dimension-based uniform convergence results , 1989, 30th Annual Symposium on Foundations of Computer Science.

[9]  Michael Frazier,et al.  Learning from a consistently ignorant teacher , 1994, COLT '94.

[10]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[11]  Nader H. Bshouty,et al.  On the exact learning of formulas in parallel , 1992, Proceedings., 33rd Annual Symposium on Foundations of Computer Science.

[12]  Yoav Freund,et al.  An improved boosting algorithm and its implications on learning complexity , 1992, COLT '92.

[13]  David Haussler,et al.  Equivalence of models for polynomial learnability , 1988, COLT '88.

[14]  Robert E. Schapire,et al.  The strength of weak learnability , 1990, Mach. Learn..

[15]  Javed A. Aslam,et al.  General bounds on statistical query learning and PAC learning with noise via hypothesis boosting , 1993, Proceedings of 1993 IEEE 34th Annual Foundations of Computer Science.

[16]  Eric B. Baum,et al.  The Perceptron Algorithm is Fast for Nonmalicious Distributions , 1990, Neural Computation.

[17]  Zhixiang Chen,et al.  On-line learning of rectangles , 1992, COLT '92.

[18]  Philip M. Long,et al.  Composite geometric concepts and polynomial predictability , 1990, COLT '90.

[19]  Zhixiang Chen,et al.  Learning unions of two rectangles in the plane with equivalence queries , 1993, COLT '93.

[20]  Zhixiang Chen,et al.  On learning discretized geometric concepts , 1994, Proceedings 35th Annual Symposium on Foundations of Computer Science.

[21]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[22]  Jeffrey Scott Vitter,et al.  Learning in parallel , 1988, COLT '88.

[23]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[24]  Herbert Edelsbrunner,et al.  Algorithms in Combinatorial Geometry , 1987, EATCS Monographs in Theoretical Computer Science.

[25]  Osamu Watanabe,et al.  An optimal parallel algorithm for learning DFA , 1994, COLT '94.

[26]  Zhixiang Chen,et al.  The Bounded Injury Priority Method and the Learnability of Unions of Rectangles , 1996, Ann. Pure Appl. Log..

[27]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[28]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[29]  Robert H. Sloan,et al.  Corrigendum to types of noise in data for concept learning , 1988, COLT '92.

[30]  Leslie G. Valiant,et al.  A general lower bound on the number of examples needed for learning , 1988, COLT '88.

[31]  Bonnie Berger,et al.  Efficient NC Algorithms for Set Cover with Applications to Learning and Geometry , 1994, J. Comput. Syst. Sci..

[32]  ERIC B. BAUM,et al.  On learning a union of half spaces , 1990, J. Complex..

[33]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1990, COLT '90.

[34]  Leslie G. Valiant,et al.  Learning Disjunction of Conjunctions , 1985, IJCAI.

[35]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[36]  Wolfgang Maass,et al.  On the complexity of learning from counterexamples , 1989, 30th Annual Symposium on Foundations of Computer Science.

[37]  Scott E. Decatur.  Statistical queries and faulty PAC oracles , 1993, COLT '93.

[38]  Wolfgang Maass,et al.  On the complexity of learning from counterexamples and membership queries , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.