Learning Nested Concept Classes with Limited Storage

Resource limitations have played an important role in the development and verification of theories about intelligent behaviour. This paper is a step towards answering the question of what effect limited memory has on the ability of intelligent machines to learn from data. Our analysis is applicable to many existing learning methods, especially those that incrementally construct a generalization by making repeated passes through a set of training data (e.g. some implementations of perceptrons, neural nets, and decision trees). Most of these methods do not store the entire training set, since they allow themselves only limited storage, a restriction that forces them to produce a compressed representation. The question we address is, how much (if any) additional processing time is required for methods with limited storage ? We measure processing time for learning algorithms by the number of passes through a data set necessary to obtain a correct generalization. Researchers have observed that for some learnin...

[1]  David W. Aha,et al.  Noise-Tolerant Instance-Based Learning Algorithms , 1989, IJCAI.

[2]  Manfred K. Warmuth,et al.  Learning nested differences of intersection-closed concept classes , 2004, Machine Learning.

[3]  Richard Cole,et al.  Parallel merge sort , 1988, 27th Annual Symposium on Foundations of Computer Science (sfcs 1986).

[4]  Ryszard S. Michalski,et al.  A theory and methodology of inductive learning , 1993 .

[5]  Manfred K. Warmuth,et al.  Learning Nested Differences of Intersection-Closed Concept Classes , 1989, COLT '89.

[6]  Z. Pylyshyn The role of location indexes in spatial perception: A sketch of the FINST spatial-index model , 1989, Cognition.

[7]  Stephen Grossberg,et al.  Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system , 1991, Neural Networks.

[8]  David P. Dobkin,et al.  Optimal Time Minimal Space Selection Algorithms , 1981, JACM.

[9]  Richard M. Karp,et al.  A Survey of Parallel Algorithms for Shared-Memory Machines , 1988 .

[10]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[11]  Steven L. Salzberg,et al.  Learning with Nested Generalized Exemplars , 1990 .

[12]  Sally Floyd,et al.  Space-bounded learning and the Vapnik-Chervonenkis dimension , 1989, COLT '89.

[13]  J. Ian Munro,et al.  Selection and sorting with limited storage , 1978, 19th Annual Symposium on Foundations of Computer Science (sfcs 1978).

[14]  Steven Salzberg,et al.  Nested Hyper-Rectangles for Exemplar-Based Learning , 1989, AII.

[15]  Alan D. Christiansen Learning to Predict in Uncertain Continuous Tasks , 1992, ML.