Comprehensiveness of tree based models: attribute dependencies and split selection

The attributes’ interdependencies have strong effect on understandability of tree based models. If strong dependencies between the attributes are not recognized and these attributes are not used as splits near the root of the tree this causes node replications in lower levels of the tree, blurs the description of dependencies and also might cause drop of accuracy. If Relief family of algorithms which is capable of estimating the attributes’ dependencies is used for split selectors we can partly overcome the problem. We describe ReliefF and RReliefF algorithms and their use in connection with tree based models. Some theoretical properties of Relief’s estimate and a recent empirical study suggest that accuracy optimization near the fringe of the tree is not necessary with these algorithms.

[1]  Marko Robnik-Sikonja,et al.  Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF , 2004, Applied Intelligence.

[2]  I. Kononenko,et al.  Linear Space Induction in First Order Logic with Relieff , 1995 .

[3]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[4]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[5]  Igor Kononenko,et al.  Estimating Attributes: Analysis and Extensions of RELIEF , 1994, ECML.

[6]  Igor Kononenko,et al.  On Biases in Estimating Multi-Valued Attributes , 1995, IJCAI.

[7]  Thomas G. Dietterich.  Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms , 1998, Neural Computation.

[8]  Marko Robnik-Sikonja,et al.  Attribute Dependencies, Understandability and Split Selection in Tree Based Models , 1999, ICML.

[9]  Carla E. Brodley,et al.  Multivariate decision trees , 2004, Machine Learning.

[10]  Steven Salzberg,et al.  On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach , 1997, Data Mining and Knowledge Discovery.

[11]  Marko Robnik-Šikonja,et al.  Modelling the effects of environmental conditions on apparent photosynthesis of Stipa bromoides by machine learning tools , 2000 .

[12]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[13]  Larry A. Rendell,et al.  A Practical Approach to Feature Selection , 1992, ML.

[14]  Carla E. Brodley,et al.  Automatic Selection of Split Criterion during Tree Growing Based on Node Location , 1995, ICML.

[15]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[16]  William H. Press,et al.  Numerical recipes in C , 2002 .

[17]  M. Robnik-Sikonja.  CORE - a system that predicts continuous variables , 1997 .

[18]  David J. Lubinsky.  Increasing the Performance and Consistency of Classification Trees by Using the Accuracy Criterion at the Leaves , 1995, ICML.

[19]  David W. Aha,et al.  A Review and Empirical Evaluation of Feature Weighting Methods for a Class of Lazy Learning Algorithms , 1997, Artificial Intelligence Review.

[20]  Marko Robnik-Sikonja,et al.  An adaptation of Relief for attribute estimation in regression , 1997, ICML.

[21]  Ivan Bratko,et al.  ASSISTANT 86: A Knowledge-Elicitation Tool for Sophisticated Users , 1987, EWSL.

[22]  Marko Robnik,et al.  Speeding up Relief algorithms with k-d trees , 2007 .