Induction as a search procedure

This chapter introduces Inductive Logic Programming (ILP) from the perspective of search algorithms in Computer Science. It first briefly considers the Version Spaces approach to induction, and then focuses on Inductive Logic Programming: from its formal definition and main techniques and strategies, to priors used to restrict the search space and optimized sequential, parallel, and stochastic algorithms. The authors hope that this presentation of the theory and applications of Inductive Logic Programming will help the reader understand the theoretical underpinnings of ILP, and also provide a helpful overview of the State-of-the-Art in the domain.

[1]  Giancarlo Ruffo,et al.  Intrusion Detection through Behavioral Data , 1999, IDA.

[2]  J. R. Quinlan.  Learning Logical Definitions from Relations , 1990 .

[3]  Fumio Mizoguchi,et al.  Parallel Execution for Speeding Up Inductive Logic Programming Systems , 1999, Discovery Science.

[4]  Luc De Raedt,et al.  Towards Learning Stochastic Logic Programs from Proof-Banks , 2005, AAAI.

[5]  Luc De Raedt,et al.  CLASSIC'CL: An Integrated ILP System , 2005, Discovery Science.

[6]  Stephen Muggleton,et al.  ILP for Mathematical Discovery , 2003, ILP.

[7]  S. Džeroski,et al.  Classification Of River Water Quality UsingMachine Learning , 1970 .

[8]  Luc De Raedt,et al.  Logical Settings for Concept-Learning , 1997, Artif. Intell..

[9]  Stephen Muggleton,et al.  Efficient Induction of Logic Programs , 1990, ALT.

[10]  Fumio Mizoguchi,et al.  Concurrent Execution of Optimal Hypothesis Search for Inverse Entailment , 2000, ILP.

[11]  James Cussens.  Bayes and Pseudo-Bayes Estimates of Conditional Probabilities and Their Reliability , 1993, ECML.

[12]  Johannes Fürnkranz,et al.  Dimensionality Reduction in ILP: A Call to Arms , 1997 .

[13]  Luc De Raedt,et al.  DLAB - A declarative language bias for concept learning and knowledge discovery engines , 1995 .

[14]  Ashwin Srinivasan,et al.  Carcinogenesis Predictions Using ILP , 1997, ILP.

[15]  Henrik Boström,et al.  Predicate Invention and Learning from Positive Examples Only , 1998, ECML.

[16]  Rui Camacho,et al.  Inducing Models of human Control Skills , 1998, ECML.

[17]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[18]  Saso Dzeroski,et al.  Induction of Slovene Nominal Paradigms , 1997, ILP.

[19]  Stasinos Konstantopoulos,et al.  A Data-Parallel Version of Aleph , 2007, ArXiv.

[20]  Ross D. King,et al.  Applying Inductive Logic Programming to Predicting Gene Function , 2004, AI Mag..

[21]  Ashwin Srinivasan,et al.  Query Transformations for Improving the Efficiency of ILP Systems , 2003, J. Mach. Learn. Res..

[22]  Stephen Muggleton,et al.  Inverse entailment and progol , 1995, New Generation Computing.

[23]  Patrick Henry Winston,et al.  Learning structural descriptions from examples , 1970 .

[24]  Jan Wielemaker,et al.  Native Preemptive Threads in SWI-Prolog , 2003, ICLP.

[25]  Hendrik Blockeel,et al.  Top-Down Induction of First Order Logical Decision Trees , 1998, AI Commun..

[26]  Luc De Raedt,et al.  Using Logical Decision Trees for Clustering , 1997, ILP.

[27]  Igor Kononenko,et al.  An Application of Ilp in a Musical Database: Learning to Compose the Two-voice Counterpoint , 1996 .

[28]  Stephen Muggleton,et al.  The Effect of Relational Background Knowledge on Learning of Protein Three-Dimensional Fold Signatures , 2001, Machine Learning.

[29]  Ashwin Srinivasan,et al.  Pharmacophore Discovery Using the Inductive Logic Programming System PROGOL , 1998, Machine Learning.

[30]  Boonserm Kijsirikul,et al.  Approximate Match of Rules Using Backpropagation Neural Networks , 2001, Machine Learning.

[31]  Michèle Sebag,et al.  Analyzing Relational Learning in the Phase Transition Framework , 2000, ICML.

[32]  James H. Graham,et al.  Accelerating the drug design process through parallel inductive logic programming data mining , 2003, Computational Systems Bioinformatics. CSB2003. Proceedings of the 2003 IEEE Bioinformatics Conference. CSB2003.

[33]  Stephen Muggleton,et al.  Machine Invention of First Order Predicates by Inverting Resolution , 1988, ML.

[34]  S. Vera,et al.  Induction of Concepts in the Predicate Calculus , 1975, IJCAI.

[35]  Rui Camacho,et al.  As lazy as it can be , 2003 .

[36]  Raymond J. Mooney,et al.  Relational Learning of Pattern-Match Rules for Information Extraction , 1999, CoNLL.

[37]  Pat Langley,et al.  Elements of Machine Learning , 1995 .

[38]  Luc De Raedt,et al.  Inductive Logic Programming: Theory and Methods , 1994, J. Log. Program..

[39]  T. Bayes.  An essay towards solving a problem in the doctrine of chances , 2003 .

[40]  Raymond J. Mooney,et al.  Using Multiple Clause Constructors in Inductive Logic Programming for Semantic Parsing , 2001, ECML.

[41]  Lubos Popelínský,et al.  Knowledge Discovery in Spatial Data by Means of ILP , 1998, PKDD.

[42]  James Cussens,et al.  Stochastic Logic Programs: Sampling, Inference and Applications , 2000, UAI.

[43]  John Nerbonne,et al.  Modeling the Phonotactic Structure of Natural Language Words with Simple Recurrent Networks , 1998 .

[44]  J. Lloyd.  Foundations of Logic Programming , 1984, Symbolic Computation.

[45]  Chris Clifton,et al.  Query flocks: a generalization of association-rule mining , 1998, SIGMOD '98.

[46]  Saso Dzeroski,et al.  Acquiring background knowledge for machine learning using function decomposition: a case study in rheumatology , 1998, Artif. Intell. Medicine.

[47]  Rajkumar Roy,et al.  Advances in Soft Computing , 2018, Lecture Notes in Computer Science.

[48]  Gavin Burnage.  Celex-a guide for users , 1990 .

[49]  Stephen Muggleton,et al.  Learning from Positive Data , 1996, Inductive Logic Programming Workshop.

[50]  Peter A. Flach,et al.  Rule Evaluation Measures: A Unifying View , 1999, ILP.

[51]  Luc De Raedt,et al.  Mining Association Rules in Multiple Relations , 1997, ILP.

[52]  Gilles Richard,et al.  A Simulated Annealing Framework for ILP , 2004, ILP.

[53]  Stephen Muggleton,et al.  Analogical Prediction , 1999, ILP.

[54]  Sašo Džeroski,et al.  Handling imperfect data in inductive logic programming , 1993 .

[55]  Pavel Brazdil,et al.  Proceedings of the European Conference on Machine Learning , 1993 .

[56]  Ehud Shapiro,et al.  Algorithmic Program Debugging , 1983 .

[57]  Alan M. Frisch,et al.  Cautious induction: An alternative to clause-at-a-time hypothesis construction in inductive logic programming , 2009, New Generation Computing.

[58]  Joseph Bockhorst,et al.  FOIL-D: Efficiently Scaling FOIL for Multi-relational Data Mining of Large Datasets , 2004, ILP.

[59]  Saso Dzeroski,et al.  Applying ILP to Diterpene Structure Elucidation from 13C NMR Spectra , 1996, Inductive Logic Programming Workshop.

[60]  Luc De Raedt,et al.  Parallel inductive logic programming , 1995 .

[61]  David H. D. Warren,et al.  Parsing as Deduction , 1983, ACL.

[62]  Ashwin Srinivasan,et al.  Mutagenesis: ILP experiments in a non-determinate biological domain , 1994 .

[63]  Johannes Fürnkranz,et al.  An Analysis of Rule Evaluation Metrics , 2003, ICML.

[64]  Stefan Wrobel,et al.  An Algorithm for Multi-relational Discovery of Subgroups , 1997, PKDD.

[65]  Nuno A. Fonseca,et al.  On Applying Tabling to Inductive Logic Programming , 2005, ECML.

[66]  Tom M. Mitchell,et al.  Learning by experimentation: acquiring and refining problem-solving heuristics , 1993 .

[67]  Akihiro Yamamoto,et al.  Inductive Logic Programming , 2003, Lecture Notes in Computer Science.

[68]  A. Kolmogorov.  Three approaches to the quantitative definition of information , 1968 .

[69]  Ashwin Srinivasan,et al.  Lattice-Search Runtime Distributions May Be Heavy-Tailed , 2002, ILP.

[70]  Charles E. Kahn,et al.  Knowledge Discovery from Structured Mammography Reports Using Inductive Logic Programming , 2005, AMIA.

[71]  J. W. Lloyd,et al.  Foundations of logic programming; (2nd extended ed.) , 1987 .

[72]  David Page,et al.  Biological applications of multi-relational data mining , 2003, SKDD.

[73]  Stan Matwin,et al.  Inductive Logic Programming , 2003, Lecture Notes in Computer Science.

[74]  Tohgoroh Matsui,et al.  Comparison of Three Parallel Implementations of an Induction Algorithm , 1998 .

[75]  Rui Carlos Camacho de Sousa Ferreira da Silva,et al.  Inducing models of human control skills using machine learning algorithms , 2000 .

[76]  Masaru Tomita,et al.  Efficient parsing for natural language , 1985 .

[77]  Alípio Mário Jorge,et al.  Architecture for Iterative Learning of Recursive Definitions , 1996 .

[78]  Hendrik Blockeel,et al.  Query Optimization in Inductive Logic Programming by Reordering Literals , 2003, ILP.

[79]  Stephen Muggleton,et al.  Searching the Subsumption Lattice by a Genetic Algorithm , 2000, ILP.

[80]  Christopher H. Bryant,et al.  Functional genomic hypothesis generation and experimentation by a robot scientist , 2004, Nature.

[81]  Nuno A. Fonseca,et al.  On Avoiding Redundancy in Inductive Logic Programming , 2004, ILP.

[82]  Jesse Davis,et al.  Establishing Identity Equivalence in Multi-Relational Domains , 2005 .

[83]  Donato Malerba,et al.  Automated acquisition of rules for document understanding , 1993, Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR '93).

[84]  Marie-Odile Cordier,et al.  Temporal abstraction and inductive logic programming for arrhythmia recognition from electrocardiograms , 2003, Artif. Intell. Medicine.

[85]  Rui Camacho,et al.  Nuno Alberto Paulino da Fonseca Parallelism in Inductive Logic Programming Systems , 2006 .

[86]  Johannes Fürnkranz,et al.  Integrative Windowing , 1998, J. Artif. Intell. Res..

[87]  Henrik Boström,et al.  Theory-Guideed Induction of Logic Programs by Inference of Regular Languages , 1996, ICML.

[88]  Katharina Morik,et al.  Combining Statistical Learning with a Knowledge-Based Approach - A Case Study in Intensive Care Monitoring , 1999, ICML.

[89]  Peter Clark,et al.  The CN2 induction algorithm , 2004, Machine Learning.

[90]  Peter Norvig,et al.  Artificial intelligence - a modern approach, 2nd Edition , 2003, Prentice Hall series in artificial intelligence.

[91]  Stephen Muggleton,et al.  Learning Structure and Parameters of Stochastic Logic Programs , 2002, ILP.

[92]  Michelangelo Ceci,et al.  Correcting the document layout: a machine learning approach , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[93]  Michèle Sebag,et al.  Relational Learning as Search in a Critical Region , 2003, J. Mach. Learn. Res..

[94]  Dietrich Wettschereck,et al.  Relational Instance-Based Learning , 1996, ICML.

[95]  Stephen G. Pulman,et al.  Experiments in Inductive Chart Parsing , 1999, Learning Language in Logic.

[96]  FREDERICK HAYES-RoTH,et al.  Schematic classification problems and their solution , 1974, Pattern Recognit..

[97]  Mark Craven,et al.  Relational Learning with Statistical Predicate Invention: Better Models for Hypertext , 2001, Machine Learning.

[98]  Marie-Odile Cordier,et al.  Application of ILP to Cardiac Arrhythmia Characterization for Chronicle Recognition , 2001, ILP.

[99]  Luc De Raedt,et al.  Clausal Discovery , 1997, Machine Learning.

[100]  Ashwin Srinivasan,et al.  A Study of Two Sampling Methods for Analyzing Large Datasets with ILP , 1999, Data Mining and Knowledge Discovery.

[101]  James Cussens,et al.  Learning Language in Logic , 2001, Lecture Notes in Computer Science.

[102]  Ross D. King,et al.  Application of Inductive Logic Programming to Structure-Based Drug Design , 2003, PKDD.

[103]  Luc De Raedt,et al.  Learning from satisfiability , 1997 .

[104]  Raymond J. Mooney,et al.  Automated refinement of first-order horn-clause domain theories , 2005, Machine Learning.

[105]  J. A. Robinson,et al.  A Machine-Oriented Logic Based on the Resolution Principle , 1965, JACM.

[106]  Gordon Plotkin,et al.  A Note on Inductive Generalization , 2008 .

[107]  G. Plotkin.  Automatic Methods of Inductive Inference , 1972 .

[108]  Luc De Raedt,et al.  Scaling Up Inductive Logic Programming by Learning from Interpretations , 1999, Data Mining and Knowledge Discovery.

[109]  Calvin Ko,et al.  Logic induction of valid behavior specifications for intrusion detection , 2000, Proceeding 2000 IEEE Symposium on Security and Privacy. S&P 2000.

[110]  Ashwin Srinivasan,et al.  Compression, Significance, and Accuracy , 1992, ML.

[111]  Ashwin Srinivasan,et al.  An Empirical Study of the Use of Relevance Information in Inductive Logic Programming , 2003, J. Mach. Learn. Res..

[112]  Luc De Raedt,et al.  First-Order jk-Clausal Theories are PAC-Learnable , 1994, Artif. Intell..

[113]  Luc De Raedt,et al.  Kernels on Prolog Proof Trees: Statistical Learning in the ILP Setting , 2006, J. Mach. Learn. Res..

[114]  Tom Michael Mitchell.  Version spaces: an approach to concept learning. , 1979 .

[115]  Eric McCreath,et al.  Extraction of Meta-Knowledge to Restrict the Hypothesis Space for ILP Systems , 1995 .

[116]  Stefan Wrobel,et al.  Relational Instance-Based Learning with Lists and Terms , 2001, Machine Learning.

[117]  Ryszard S. Michalski,et al.  AQVAL/1--Computer Implementation of a Variable-Valued Logic System VL1 and Examples of its Application to Pattern Recognition , 1973, IJCAI 1973.

[118]  James Cussens.  Part-of-Speech Tagging Using Progol , 1997, ILP.

[119]  Luc De Raedt,et al.  Basic Principles of Learning Bayesian Logic Programs , 2008, Probabilistic Inductive Logic Programming.

[120]  Saso Dzeroski,et al.  Hierarchical Multi-classification with Predictive Clustering Trees in Functional Genomics , 2005, EPIA.

[121]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[122]  Bojan Cestnik,et al.  Estimating Probabilities: A Crucial Task in Machine Learning , 1990, ECAI.

[123]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[124]  Michèle Sebag,et al.  Tractable Induction and Classification in First Order Logic Via Stochastic Matching , 1997, IJCAI.

[125]  Donato Malerba,et al.  On the Effect of Caching in Recursive Theory Learning , 2004, ILP.

[126]  Luc De Raedt,et al.  Proceedings of the 12th European Conference on Machine Learning , 2001 .

[127]  Luc De Raedt,et al.  Bayesian Logic Programs , 2001, ILP Work-in-progress reports.

[128]  Jude W. Shavlik,et al.  Learning an Approximation to Inductive Logic Programming Clause Evaluation , 2004, ILP.

[129]  Rui Camacho.  Improving the efficiency of ILP systems using an Incremental Language Level Search , 2002 .

[130]  Gordon Plotkin,et al.  A Further Note on Inductive Generalization , 2008 .

[131]  Jan Struyf,et al.  Techniques for Improving the Efficiency of Inductive Logic Programming in the Context of Data Mining (Technieken voor het verhogen van de efficiëntie van inductief logisch programmeren in de context van datamining) , 2004 .

[132]  Ashwin Srinivasan,et al.  ILP: A Short Look Back and a Longer Look Forward , 2003, J. Mach. Learn. Res..

[133]  Céline Rouveirol,et al.  Proceedings of the 10th European Conference on Machine Learning , 1998 .

[134]  Saso Dzeroski,et al.  Inducing Polynomial Equations for Regression , 2004, ECML.

[135]  Ashwin Srinivasan,et al.  A Note on Two Simple Transformations for Improving the Efficiency of an ILP System , 2000, ILP.

[136]  G. Booij.  The Phonology of Dutch , 1995 .

[137]  James Cussens,et al.  CLP(BN): Constraint Logic Programming for Probabilistic Knowledge , 2002, Probabilistic Inductive Logic Programming.

[138]  Gerhard Widmer,et al.  Relational IBL in Music with a New Structural Similarity Measure , 2003, ILP.

[139]  David Page,et al.  Mode Directed Path Finding , 2005, ECML.

[140]  Stephen Muggleton,et al.  A Strategy for Constructing New Predicates in First-Order Logic , 1988, EWSL.

[141]  Stephen Muggleton,et al.  Combining Inductive Logic Programming, Active Learning and Robotics to Discover the Function of Genes , 2001, Electron. Trans. Artif. Intell..

[142]  John Nerbonne,et al.  Phonotactics in Inductive Logic Programming , 2004, Intelligent Information Systems.

[143]  Raymond J. Mooney,et al.  Learning to Parse Database Queries Using Inductive Logic Programming , 1996, AAAI/IAAI, Vol. 2.

[144]  Saso Dzeroski,et al.  Experiments in Predicting Biodegradability , 1999, ILP.

[145]  Saso Dzeroski,et al.  Proceedings of the 7th International Workshop on Inductive Logic Programming , 1997 .

[146]  William W. Cohen.  Grammatically Biased Learning: Learning Logic Programs Using an Explicit Antecedent Description Language , 1994, Artif. Intell..

[147]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[148]  Jude W. Shavlik,et al.  Learning Ensembles of First-Order Clauses for Recall-Precision Curves: A Case Study in Biomedical Information Extraction , 2004, ILP.

[149]  Bart Demoen,et al.  Improving the Efficiency of Inductive Logic Programming Through the Use of Query Packs , 2011, J. Artif. Intell. Res..

[150]  Henrik Boström.  Induction of Recursive Transfer Rules , 1999, Learning Language in Logic.

[151]  John Nerbonne,et al.  Learning the Logic of Simple Phonotactics , 1999, Learning Language in Logic.

[152]  Saso Dzeroski,et al.  ILP Experiments in Detecting Traffic Problems , 1998, ECML.