Separate-and-Conquer Rule Learning

This paper is a survey of inductive rule learning algorithms that use a separate-and-conquer strategy. This strategy can be traced back to the AQ learning system and still enjoys popularity as can be seen from its frequent use in inductive logic programming systems. We will put this wide variety of algorithms into a single framework and analyze them along three different dimensions, namely their search, language and overfitting avoidance biases.

[1]  Nada Lavrac,et al.  The Multi-Purpose Incremental Learning System AQ15 and Its Testing Application to Three Medical Domains , 1986, AAAI.

[2]  D. Haussler,et al.  Boolean Feature Discovery in Empirical Learning , 1990, Machine Learning.

[3]  Maurice Bruynooghe,et al.  Declarative bias for specific-to-general ILP systems , 1995, Machine Learning.

[4]  Ryszard S. Michalski,et al.  Hypothesis-Driven Constructive Induction in AQ17-HCI: A Method and Experiments , 1994, Machine Learning.

[5]  H. Theron,et al.  BEXA: A Covering Algorithm for Learning Propositional Concept Descriptions , 1996, Machine Learning.

[6]  Bernhard Pfahringer,et al.  Controlling Constructive Induction in CIPF: An MDL Approach , 1994, ECML.

[7]  Gerhard Widmer.  Combining Knowledge-Based and Instance-Based Learning to Exploit Qualitative Knowledge , 1993, Informatica.

[8]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[9]  R. Michalski,et al.  Multistrategy Constructive Induction: AQ17-MCI , 1993 .

[10]  Francesco Bergadano,et al.  An Interactive System to Learn Functional Logic Programs , 1993, IJCAI.

[11]  Tony R. Martinez,et al.  The BBG Rule Induction Algorithm , 1993 .

[12]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[13]  J. W. Lloyd,et al.  Foundations of logic programming; (2nd extended ed.) , 1987 .

[14]  Raymond J. Mooney,et al.  Combining Top-down and Bottom-up Techniques in Inductive Logic Programming , 1994, ICML.

[15]  Dunja Mladenic,et al.  Combinatorial Optimization in Inductive Concept Learning , 1993, ICML.

[16]  Tom M. Mitchell,et al.  Generalization as Search , 2002 .

[17]  Michael J. Pazzani,et al.  An Investigation of Noise-Tolerant Relational Concept Learning Algorithms , 1991, ML.

[18]  Ryszard S. Michalski,et al.  A Theory and Methodology of Inductive Learning , 1983, Artificial Intelligence.

[19]  Stefan Kramer.  CN2-MCI: A Two-Step Method for Constructive Induction , 1994 .

[20]  R. Mike Cameron-Jones,et al.  Avoiding Pitfalls When Learning Recursive Theories , 1993, IJCAI.

[21]  Johannes Fürnkranz.  Top-Down Pruning in Relational Learning , 1994, ECAI.

[22]  Markus Wiese,et al.  A bidirectional ILP algorithm , 1996 .

[23]  C. S. Wallace,et al.  An Information Measure for Classification , 1968, Comput. J..

[24]  J. Ross Quinlan,et al.  Determinate Literals in Inductive Logic Programming , 1991, IJCAI.

[25]  J. Ross Quinlan,et al.  The Minimum Description Length Principle and Categorical Theories , 1994, ICML.

[26]  Henrik Boström,et al.  Covering vs. Divide-and-Conquer for Top-Down Induction of Logic Programs , 1995, IJCAI.

[27]  J. Ross Quinlan,et al.  Simplifying decision trees , 1987, Int. J. Hum. Comput. Stud..

[28]  Pedro M. Domingos.  Linear-Time Rule Induction , 1996, KDD.

[29]  Claire Nédellec,et al.  Declarative Bias in ILP , 1996 .

[30]  L. D. Raedt.  Interactive theory revision: an inductive logic programming approach , 1992 .

[31]  S. Matwin,et al.  Learning Two-Tiered Descriptions of Flexible Concepts: The POSEIDON System , 1992, Machine Learning.

[32]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[33]  William W. Cohen.  Grammatically Biased Learning: Learning Logic Programs Using an Explicit Antecedent Description Language , 1994, Artif. Intell..

[34]  Francesco Bergadano,et al.  Deduction in Top-Down Inductive Learning , 1989, ML.

[35]  R. Mike Cameron-Jones,et al.  Oversearching and Layered Search in Empirical Learning , 1995, IJCAI.

[36]  Michael J. Pazzani,et al.  Reducing Misclassification Costs , 1994, ICML.

[37]  Pankaj Mehra,et al.  Constructive Induction Framework , 1989, ML Workshop.

[38]  Johannes Fürnkranz,et al.  Pruning Algorithms for Rule Learning , 1997, Machine Learning.

[39]  Francesco Bergadano,et al.  A Knowledge Intensive Approach to Concept Induction , 1988, ML Workshop.

[40]  Wray L. Buntine,et al.  A further comparison of splitting rules for decision-tree induction , 2004, Machine Learning.

[41]  Marco Botta,et al.  Comparison of Search Strategies in Learning Relations , 1992, ECAI.

[42]  Gilles Venturini,et al.  SIA: A Supervised Inductive Algorithm with Genetic Search for Learning Attributes based Concepts , 1993, ECML.

[43]  Ron Kohavi,et al.  Wrappers for performance enhancement and oblivious decision graphs , 1995 .

[44]  De Raedt,et al.  Advances in Inductive Logic Programming , 1996 .

[45]  Igor Kononenko,et al.  SFOIL: Stochastic Approach to Inductive Logic Programming , 1993 .

[46]  Jorg-uwe Kietz,et al.  Controlling the Complexity of Learning in Logic through Syntactic and Task-Oriented Models , 1992 .

[47]  Michael Bain,et al.  Experiments in Non-Monotonic Learning , 1991, ML.

[48]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[49]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[50]  Ivan Bratko,et al.  First Order Regression , 1997, Machine Learning.

[51]  Johannes Fürnkranz,et al.  Incremental Reduced Error Pruning , 1994, ICML.

[52]  M. Pazzani,et al.  The Utility of Knowledge in Inductive Learning , 1992, Machine Learning.

[53]  Lorenza Saitta,et al.  Pattern Recognition and Valiant's Learning Framework , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[54]  Geoffrey I. Webb.  Learning Decision Lists by Prepending Inferred Rules , 2005 .

[55]  Luc De Raedt,et al.  Declarative Bias for Specific-to-General ILP Systems , 1994, Machine Learning.

[56]  Attilio Giordana,et al.  Learning Structured Concepts Using Genetic Algorithms , 1992, ML.

[57]  Céline Rouveirol.  Flattening and Saturation: Two Representation Changes for Generalization , 2004, Machine Learning.

[58]  Matevz Kovacic.  Stochastic Inductive Logic Programming , 1994 .

[59]  Geoffrey I. Webb.  OPUS: An Efficient Admissible Algorithm for Unordered Search , 1995, J. Artif. Intell. Res..

[60]  Ron Kohavi,et al.  Automatic Parameter Selection by Minimizing Estimated Error , 1995, ICML.

[61]  Luc De Raedt,et al.  Inductive Constraint Logic , 1995, ALT.

[62]  Michael J. Pazzani,et al.  HYDRA: A Noise-tolerant Relational Concept Learning Algorithm , 1993, IJCAI.

[63]  Oren Etzioni,et al.  Learning Decision Lists Using Homogeneous Rules , 1994, AAAI.

[64]  Geoffrey I. Webb.  Recent Progress in Learning Decision Lists by Prepending Inferred Rules , 1994 .

[65]  Sholom M. Weiss,et al.  Optimized rule induction , 1993, IEEE Expert.

[66]  Stefan Kramer,et al.  Structural Regression Trees , 1996, AAAI/IAAI, Vol. 1.

[67]  Larry A. Rendell,et al.  Learning Structural Decision Trees from Examples , 1991, IJCAI.

[68]  Saso Dzeroski,et al.  Learning Nonrecursive Definitions of Relations with LINUS , 1991, EWSL.

[69]  Luc De Raedt,et al.  DLAB: A Declarative Language Bias Formalism , 1996, ISMIS.

[70]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[71]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[72]  J. Ross Quinlan.  Learning First-Order Definitions of Functions , 1996, J. Artif. Intell. Res..

[73]  Luc De Raedt,et al.  Top-down induction of logical decision trees , 1997 .

[74]  John Mingers,et al.  An Empirical Comparison of Selection Measures for Decision-Tree Induction , 1989, Machine Learning.

[75]  J. Ross Quinlan,et al.  Learning Efficient Classification Procedures and Their Application to Chess End Games , 1983 .

[76]  William W. Cohen.  Fast Eeective Rule Induction , 1995 .

[77]  Peter Clark,et al.  The CN2 Induction Algorithm , 1989, Machine Learning.

[78]  M. Kovacic.  Markovian neural networks , 1991, Biological Cybernetics.

[79]  B. Pfahringer.  Practical Uses of the Minimum Description Length Principle in Inductive Learning , 1995 .

[80]  Padhraic Smyth,et al.  Information-Theoretic Rule Induction , 1988, ECAI.

[81]  Nicolas Helft,et al.  Induction as Nonmonotonic Inference , 1989, KR.

[82]  Wray L. Buntine,et al.  A Further Comparison of Splitting Rules for Decision-Tree Induction , 1992, Machine Learning.

[83]  Sholom M. Weiss,et al.  Rule-Based Regression , 1993, IJCAI.

[84]  Stephen Muggleton,et al.  Efficient Induction of Logic Programs , 1990, ALT.

[85]  Bernhard Pfahringer.  Robust Constructive Induction , 1994, KI.

[86]  Bernhard Pfahringer.  A New MDL Measure for Robust Rule Induction (Extended Abstract) , 1995, ECML.

[87]  R. Mike Cameron-Jones,et al.  The Complexity of Batch Approaches to Reduced Error Rule Set Induction , 1996, PRICAI.

[88]  Lorenza Saitta,et al.  Automated Concept Acquisition in Noisy Environments , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[89]  Marco Botta,et al.  SMART+: A Multi-Strategy Learning Tool , 1993, IJCAI.

[90]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[91]  Sholom M. Weiss,et al.  Rule-based Machine Learning Methods for Functional Prediction , 1995, J. Artif. Intell. Res..

[92]  Pedro M. Domingos.  Unifying Instance-Based and Rule-Based Induction , 1996, Machine Learning.

[93]  Jadzia Cendrowska,et al.  PRISM: An Algorithm for Inducing Modular Rules , 1987, Int. J. Man Mach. Stud..

[94]  Saso Dzeroski,et al.  Applying ILP to Diterpene Structure Elucidation from 13C NMR Spectra , 1996, Inductive Logic Programming Workshop.

[95]  H. Simon,et al.  Heuristics for empirical discovery , 1987 .

[96]  Igor Kononenko,et al.  Learning as Optimization: Stochastic Generation of Multiple Knowledge , 1992, ML.

[97]  Dieter Fensel,et al.  From JoJo to Frog: Extending a bi-directional Search Strategy to a more flexible three-directional Search , 1999 .

[98]  Francesco Bergadano,et al.  Inductive Logic Programming: From Machine Learning to Software Engineering , 1995 .

[99]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[100]  Masayuki Numao,et al.  Discrimination-Based Constructive Induction of Logic Programs , 1992, AAAI.

[101]  Dieter Fensel,et al.  Refinement of Rule Sets with JoJo , 1993, ECML.

[102]  Ryszard S. Michalski,et al.  AQVAL/1--Computer Implementation of a Variable-Valued Logic System VL1 and Examples of its Application to Pattern Recognition , 1973, IJCAI 1973.

[103]  D. Wolpert.  On Overfitting Avoidance as Bias , 1993 .

[104]  John Wylie Lloyd,et al.  Foundations of Logic Programming , 1987, Symbolic Computation.

[105]  William W. Cohen.  Efficient Pruning Methods for Separate-and-Conquer Rule Learning Systems , 1993, IJCAI.

[106]  Geoffrey I. Webb,et al.  Inducing diagnostic rules for glomerular disease with the DLG machine learning algorithm , 1992, Artif. Intell. Medicine.

[107]  Johannes Fürnkranz.  A Tight Integration of Pruning and Learning (Extended Abstract) , 1995, ECML.

[108]  Michael J. Pazzani,et al.  Relational Clichés: Constraining Induction During Relational Learning , 1991, ML.

[109]  Paul E. Utgoff,et al.  Shift of bias for inductive concept learning , 1984 .

[110]  William W. Cohen.  Fast Effective Rule Induction , 1995, ICML.

[111]  Ryszard S. Michalski,et al.  CONSTRUCTIVE INDUCTION FROM DATA IN AQ17-DCI: Further Experiments , 1991 .

[112]  Céline Rouveirol,et al.  Extensions of Inversion of Resolution Applied to Theory Completion , 1992 .

[113]  Robert C. Holte,et al.  Concept Learning and the Problem of Small Disjuncts , 1989, IJCAI.

[114]  Kurzfassung Diese,et al.  Eecient Pruning Methods for Relational Learning , 1994 .

[115]  Ehud Y. Shapiro.  An Algorithm that Infers Theories from Facts , 1981, IJCAI.

[116]  Raymond J. Mooney,et al.  Induction of First-Order Decision Lists: Results on Learning the Past Tense of English Verbs , 1995, J. Artif. Intell. Res..

[117]  Peter Clark,et al.  Rule Induction with CN2: Some Recent Improvements , 1991, EWSL.

[118]  J. Ross Quinlan,et al.  Generating Production Rules from Decision Trees , 1987, IJCAI.

[119]  Johannes Fürnkranz.  Efficient Pruning Methods for Relational Learning , 1994 .

[120]  Cullen Schaffer.  Overfitting avoidance as bias , 2004, Machine Learning.

[121]  V. Vapnik,et al.  Necessary and Sufficient Conditions for the Uniform Convergence of Means to their Expectations , 1982 .

[122]  Stephen Muggleton,et al.  An Experimental Comparison of Human and Machine Learning Formalisms , 1989, ML.

[123]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[124]  Raymond J. Mooney,et al.  Encouraging experimental results on learning CNF , 1995, Machine Learning.

[125]  Sholom M. Weiss,et al.  Reduced Complexity Rule Induction , 1991, IJCAI.

[126]  Luc De Raedt,et al.  Indirect relevance and bias in inductive concept-learning , 1990 .

[127]  Masayuki Numao,et al.  Efficient Learning of Logic Programs with Non-determinant, Non-discriminating Literals , 1991, ML.

[128]  Johannes Fürnkranz,et al.  FOSSIL: A Robust Relational Learner , 1994, ECML.

[129]  J. Ross Quinlan,et al.  Learning logical definitions from relations , 1990, Machine Learning.