Instance-Based Learning Algorithms

Storing and using specific instances improves the performance of several supervised learning algorithms. These include algorithms that learn decision trees, classification rules, and distributed networks. However, no investigation has analyzed algorithms that use only specific instances to solve incremental learning tasks. In this paper, we describe a framework and methodology, called instance-based learning, that generates classification predictions using only specific instances. Instance-based learning algorithms do not maintain a set of abstractions derived from specific instances. This approach extends the nearest neighbor algorithm, which has large storage requirements. We describe how storage requirements can be significantly reduced with, at most, minor sacrifices in learning rate and classification accuracy. While the storage-reducing algorithm performs well on several real-world databases, its performance degrades rapidly with the level of attribute noise in training instances. Therefore, we extended it with a significance test to distinguish noisy instances. This extended algorithm's performance degrades gracefully with increasing noise levels and compares favorably with a noise-tolerant decision tree algorithm.

[1]  M. Kendall.  Probability and Statistical Inference , 1956, Nature.

[2]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[3]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[4]  C. G. Hilborn,et al.  The Condensed Nearest Neighbor Rule , 1967 .

[5]  Peter E. Hart,et al.  The condensed nearest neighbor rule (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[6]  G. Gates,et al.  The reduced nearest neighbor rule (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[7]  Michael Ian Shamos,et al.  Closest-point problems , 1975, 16th Annual Symposium on Foundations of Computer Science (sfcs 1975).

[8]  Douglas L. Medin,et al.  Context theory of classification learning. , 1978 .

[9]  P. J. Green,et al.  Probability and Statistical Inference , 1978 .

[10]  Ryszard S. Michalski,et al.  Selection of Most Representative Training Examples and Incremental Generation of VL1 Hypotheses: The Underlying Methodology and the Description of Programs ESEL and AQ11 , 1978 .

[11]  Robert V. Hogg,et al.  Probability and Statistical Inference , 1978, An R Companion for the Third Edition of The Fundamentals of Political Science Research.

[12]  Belur V. Dasarathy,et al.  Nosing Around the Neighborhood: A New System Structure and Classification Rule for Recognition in Partially Exposed Environments , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Thomas G. Dietterich,et al.  A Comparative Review of Selected Methods for Learning from Examples , 1983 .

[14]  L. Barsalou,et al.  Ad hoc categories , 1983, Memory & cognition.

[15]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[16]  Douglas L. Hintzman,et al.  "Schema Abstraction" in a Multiple-Trace Memory Model , 1986 .

[17]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[18]  David Haussler,et al.  Classifying learnable geometric concepts with the Vapnik-Chervonenkis dimension , 1986, STOC '86.

[19]  Douglas H. Fisher,et al.  A Case Study of Incremental Concept Induction , 1986, AAAI.

[20]  R. Nosofsky.  Attention, similarity, and the identification-categorization relationship. , 1986, Journal of experimental psychology. General.

[21]  Nada Lavrac,et al.  The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains , 1986, AAAI.

[22]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[23]  Ivan Bratko,et al.  ASSISTANT 86: A Knowledge-Elicitation Tool for Sophisticated Users , 1987, EWSL.

[24]  E. R. Bareiss,et al.  Protos: An Exemplar-Based Learning Apprentice1 , 1987 .

[25]  Paul Compton,et al.  Inductive knowledge acquisition: a case study , 1987 .

[26]  Gary L. Bradshaw,et al.  Learning about speech sounds: The NEXUS Project , 1987 .

[27]  J. Ross Quinlan,et al.  Generating Production Rules from Decision Trees , 1987, IJCAI.

[28]  J. Ross Quinlan,et al.  An Empirical Comparison of Genetic and Decision-Tree Classifiers , 1988, ML.

[29]  K. Jabbour,et al.  ALFA: automated load forecasting assistant , 1988 .

[30]  Ray Bareiss,et al.  Protos: An Exemplar-Based Learning Apprentice , 1988, Int. J. Man Mach. Stud..

[31]  Larry A. Rendell.  Learning Hard Concepts , 1988, EWSL.

[32]  Phyllis Koton,et al.  Reasoning about Evidence in Causal Explanations , 1988, AAAI.

[33]  Shaul Markovitch,et al.  Information Filters and Their Implementation in the SYLLOG System , 1989, ML.

[34]  D. Benjamin.  Change of Representation and Inductive Bias , 1989 .

[35]  David W. Aha,et al.  Incremental, Instance-Based Learning of Independent and Graded Concept Descriptions , 1989, ML.

[36]  David W. Aha,et al.  Noise-Tolerant Instance-Based Learning Algorithms , 1989, IJCAI.

[37]  IDL, or Taming the Multiplexer. , 1989 .

[38]  David W. Aha,et al.  Instance‐based prediction of real‐valued attributes , 1989, Comput. Intell..

[39]  R. Detrano,et al.  International application of a new probability algorithm for the diagnosis of coronary artery disease. , 1989, The American journal of cardiology.

[40]  Peter Clark.  Exemplar-Based Reasoning in Geological Prospect Appraisal , 1989 .

[41]  E. R. Bareiss,et al.  Protos: An Exemplar-Based Learning Apprentice , 1988, Int. J. Man Mach. Stud..

[42]  David W. Aha,et al.  Comparing Instance-Averaging with Instance-Saving Learning Algorithms , 1990 .

[43]  G. Kane.  Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol 1: Foundations, vol 2: Psychological and Biological Models , 1994 .

[44]  G. Gates.  The Reduced Nearest Neighbor Rule , 1998 .

[45]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[46]  Karin Ackermann,et al.  Categories and Concepts , 2003, Job 28. Cognition in Context.

[47]  Paul E. Utgoff,et al.  Incremental Induction of Decision Trees , 1989, Machine Learning.

[48]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[49]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[50]  S. Hampson,et al.  Learning and using specific instances , 2004, Biological Cybernetics.

[51]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.