Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF

Current inductive machine learning algorithms typically use greedy search with limited lookahead. This prevents them to detect significant conditional dependencies between the attributes that describe training objects. Instead of myopic impurity functions and lookahead, we propose to use RELIEFF, an extension of RELIEF developed by Kira and Rendell [10, 11], for heuristic guidance of inductive learning algorithms. We have reimplemented Assistant, a system for top down induction of decision trees, using RELIEFF as an estimator of attributes at each selection step. The algorithm is tested on several artificial and several real world problems and the results are compared with some other well known machine learning algorithms. Excellent results on artificial data sets and two real world problems show the advantage of the presented approach to inductive learning.

[1]  Philip J. Stone,et al.  Experiments in induction , 1966 .

[2]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[3]  I. Bratko,et al.  Learning decision rules in noisy domains , 1987 .

[4]  Ivan Bratko,et al.  ASSISTANT 86: A Knowledge-Elicitation Tool for Sophisticated Users , 1987, EWSL.

[5]  Larry A. Rendell,et al.  Constructive Induction On Decision Trees , 1989, IJCAI.

[6]  Padhraic Smyth,et al.  A Hybrid Rule-Based/Bayesian Classifier , 1990, ECAI.

[7]  Bojan Cestnik,et al.  Estimating Probabilities: A Crucial Task in Machine Learning , 1990, ECAI.

[8]  Ivan Bratko,et al.  On Estimating Probabilities in Tree Pruning , 1991, EWSL.

[9]  Padhraic Smyth,et al.  Rule Induction Using Information Theory , 1991, Knowledge Discovery in Databases.

[10]  Bojan Dolsak,et al.  The Application of Inductive Logic Programming to Finite Element Mesh Design , 1992 .

[11]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[12]  Larry A. Rendell,et al.  The Feature Selection Problem: Traditional Methods and a New Algorithm , 1992, AAAI.

[13]  Larry A. Rendell,et al.  Lookahead Feature Construction for Learning Hard Concepts , 1993, International Conference on Machine Learning.

[14]  Igor Kononenko,et al.  Inductive and Bayesian learning in medical diagnosis , 1993, Appl. Artif. Intell..

[15]  Igor Kononenko,et al.  SFOIL: Stochastic Approach to Inductive Logic Programming , 1993 .

[16]  Dunja Mladenic,et al.  Combinatorial Optimization in Inductive Concept Learning , 1993, ICML.

[17]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[18]  J. Ross Quinlan,et al.  The Minimum Description Length Principle and Categorical Theories , 1994, ICML.

[19]  Bill Fulkerson,et al.  Machine Learning, Neural and Statistical Classification , 1995 .

[20]  Igor Kononenko,et al.  On Biases in Estimating Multi-Valued Attributes , 1995, IJCAI.

[21]  I. Kononenko,et al.  Linear Space Induction in First Order Logic with Relieff , 1995 .

[22]  Se June Hong,et al.  Use of Contextaul Information for Feature Ranking and Discretization , 1997, IEEE Trans. Knowl. Data Eng..

[23]  A. Gosavi,et al.  General Statistics , 2000, 2018 Inland Transport Statistics for Europe and North America.

[24]  R. Ichise,et al.  Inductive Logic Programming and Genetic Programming , 1998, ECAI.

[25]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[26]  Kevin Barraclough,et al.  I and i , 2001, BMJ : British Medical Journal.

[27]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[28]  I. Bratko,et al.  Information-based evaluation criterion for classifier's performance , 2004, Machine Learning.

[29]  Ivan Bratko,et al.  Information-Based Evaluation Criterion for Classifier's Performance , 1991, Machine Learning.