Unifying Instance-Based and Rule-Based Induction

Several well-developed approaches to inductive learning low exist, but each has specific limitations that are hard to overcome. Multi-strategy learning attempts to tackle this problem combining multiple methods in one algorithm. This article describes a unification of two widely-used empirical approaches: rule induction and instance-based learning. In the new algorithm, instances are treated as maximally specific rules, and classification is oerformed using a best-match strategy. Rules are learned by gradually generalizing instances until no improvement in apparent accuracy is obtained. Theoretical analysis shows this approach to be efficient. It is implemented in the RISE 3.1 system. In an extensive empirical study, RISE consistently achieves higher accuracies than state-of-the-art representatives of both its parent approaches (PEBLS and CN2), as well as a decision tree learner (C4.5). Lesion studies show that eachoof RISE's components is essential to this performance. Most significantly, in 14 of the 30 domains studied, RISE is more accurate than the best of PEBLS and CN2, showing that a significant synergy can be obtained by combining multiple empirical methods.

[1]  Pedro M. Domingos.  The RISE system: conquering without separating , 1994, Proceedings Sixth International Conference on Tools with Artificial Intelligence. TAI 94.

[2]  Pedro M. Domingos.  Two-way induction , 1995, Proceedings of 7th IEEE International Conference on Tools with Artificial Intelligence.

[3]  Lawrence Davis,et al.  A Hybrid Genetic Algorithm for Classification , 1991, IJCAI.

[4]  Thomas G. Dietterich,et al.  An experimental comparison of the nearest-neighbor and nearest-hyperrectangle algorithms , 1995, Machine Learning.

[5]  Paul S. Rosenbloom,et al.  Improving Rule-Based Systems Through Case-Based Reasoning , 1991, AAAI.

[6]  Yoram Biberman,et al.  A Context Similarity Measure , 1994, ECML.

[7]  J. Ross Quinlan,et al.  Generating Production Rules from Decision Trees , 1987, IJCAI.

[8]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[9]  Cullen Schaffer,et al.  A Conservation Law for Generalization Performance , 1994, ICML.

[10]  J. Ross Quinlan,et al.  Combining Instance-Based and Model-Based Learning , 1993, ICML.

[11]  David W. Aha,et al.  Feature Selection for Case-Based Classification of Cloud Types: An Empirical Comparison , 1994 .

[12]  Cullen Schaffer.  Cross-Validation, Stacking and Bi-Level Stacking: Meta-Methods for Classification Learning , 1994 .

[13]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[14]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[15]  Tim Niblett,et al.  Constructing Decision Trees in Noisy Domains , 1987, EWSL.

[16]  Dietrich Wettschereck,et al.  A Hybrid Nearest-Neighbor and Nearest-Hyperrectangle Algorithm , 1994, ECML.

[17]  Ryszard S. Michalski,et al.  A theory and methodology of inductive learning , 1993 .

[18]  Jason Catlett,et al.  Megainduction: A Test Flight , 1991, ML.

[19]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[20]  D.E. Goldberg,et al.  Classifier Systems and Genetic Algorithms , 1989, Artif. Intell..

[21]  Paul E. Utgoff,et al.  Perceptron Trees : A Case Study in ybrid Concept epresentations , 1999 .

[22]  Janusz Wnek,et al.  Hypothesis-driven constructive induction , 1993 .

[23]  Padhraic Smyth,et al.  A Hybrid Rule-Based/Bayesian Classifier , 1990, ECAI.

[24]  Robert C. Holte,et al.  Concept Learning and the Problem of Small Disjuncts , 1989, IJCAI.

[25]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[26]  Peter Clark,et al.  Rule Induction with CN2: Some Recent Improvements , 1991, EWSL.

[27]  Nada Lavrac,et al.  The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains , 1986, AAAI.

[28]  Kai Ming Ting,et al.  Discretization of Continuous-Valued Attributes and Instance-Based Learning , 1994 .

[29]  Alberto L. Sangiovanni-Vincentelli,et al.  Inferring Reduced Ordered Decision Graphs of Minimum Description Length , 1995, ICML.

[30]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[31]  Wray L. Buntine.  Learning Classification Rules Using Bayes , 1989, ML.

[32]  Hidehiko Tanaka,et al.  An Optimal Weighting Criterion of Case Indexing for Both Numeric and Symbolic Attributes , 1994 .

[33]  J. R. Quinlan.  Learning Logical Definitions from Relations , 1990 .

[34]  Ron Kohavi,et al.  Oblivious Decision Trees, Graphs, and Top-Down Pruning , 1995, IJCAI.

[35]  W. Spears,et al.  For Every Generalization Action, Is There Really an Equal and Opposite Reaction? , 1995, ICML.

[36]  David W. Aha,et al.  A study of instance-based algorithms for supervised learning tasks: mathematical, empirical, and psychological evaluations , 1990 .

[37]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[38]  Pedro M. Domingos.  Rule Induction and Instance-Based Learning: A Unified Approach , 1995, IJCAI.

[39]  Alexander G. Gray,et al.  Retrofitting Decision Tree Classifiers Using Kernel Density Estimation , 1995, ICML.

[40]  Christopher K. Riesbeck,et al.  Inside Case-Based Reasoning , 1989 .

[41]  Pedro M. Domingos,et al.  The RISE 2.0 System: A Case Study in Multistrategy Learning , 1995 .

[42]  Raymond J. Mooney,et al.  Theory Refinement Combining Analytical and Empirical Methods , 1994, Artif. Intell..

[43]  Paul D. Scott,et al.  Why Generalize? Hybrid Representations and Instance-Based Learning , 1992, ECAI.

[44]  J. Zhang,et al.  A method that combines inductive learning with exemplar-based learning , 1990, [1990] Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence.

[45]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[46]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[47]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.