Comparison of Greedy Algorithms for Decision Tree Optimization

This chapter is devoted to the study of 16 types of greedy algorithms for decision tree construction. The dynamic programming approach is used for construction of optimal decision trees. Optimization is performed relative to minimal values of average depth, depth, number of nodes, number of terminal nodes, and number of nonterminal nodes of decision trees. We compare average depth, depth, number of nodes, number of terminal nodes and number of nonterminal nodes of constructed trees with minimum values of the considered parameters obtained based on a dynamic programming approach. We report experiments performed on data sets from UCI ML Repository and randomly generated binary decision tables. As a result, for depth, average depth, and number of nodes we propose a number of good heuristics.

[1]  Urszula Boryczka,et al.  Ant Colony Decision Trees - A New Method for Constructing Decision Trees Based on Ant Colony Optimization , 2010, ICCCI.

[2]  Kenneth C. Sevcik,et al.  The synthetic approach to decision table conversion , 1976, CACM.

[3]  Igor Kononenko,et al.  On Biases in Estimating Multi-Valued Attributes , 1995, IJCAI.

[4]  Guoyin Wang,et al.  Rough Sets, Fuzzy Sets, Data Mining, and Granular Computing , 2013, Lecture Notes in Computer Science.

[5]  Oren Etzioni,et al.  Representation design and brute-force induction in a Boeing manufacturing domain , 1994, Appl. Artif. Intell..

[6]  Ronald L. Rivest,et al.  Constructing Optimal Binary Decision Trees is NP-Complete , 1976, Inf. Process. Lett..

[7]  Igor Chikalov,et al.  Comparison of Greedy Algorithms for Decision Tree Construction , 2011, KDIR.

[8]  Tadeusz M. Szuba,et al.  Computational Collective Intelligence , 2001, Lecture Notes in Computer Science.

[9]  Andrzej Skowron,et al.  Transactions on Rough Sets IV , 2005, Trans. Rough Sets.

[10]  M. Garey.  Optimal Binary Identification Procedures , 1972 .

[11]  Igor Chikalov,et al.  Relationships between Depth and Number of Misclassifications for Decision Trees , 2011, RSFDGrC.

[12]  Usama M. Fayyad,et al.  The Attribute Selection Problem in Decision Tree Generation , 1992, AAAI.

[13]  Gabriela Lindemann,et al.  Concurrency Specification and Programming , 2005 .

[14]  Igor Chikalov,et al.  On Optimization of Decision Trees , 2005, Trans. Rough Sets.

[15]  Sadaaki Miyamoto,et al.  Rough Sets and Current Trends in Computing , 2012, Lecture Notes in Computer Science.

[16]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[17]  Simon Kasif,et al.  Induction of Oblique Decision Trees , 1993, IJCAI.

[18]  Z. Pawlak.  Rough Sets: Theoretical Aspects of Reasoning about Data , 1991 .

[19]  Igor Chikalov,et al.  Consecutive Optimization of Decision Trees Concerning Various Complexity Measures , 2004, Fundam. Informaticae.

[20]  Mikhail Moshkov,et al.  Time Complexity of Decision Trees , 2005, Trans. Rough Sets.

[21]  Igor Chikalov,et al.  On Algorithm for Building of Optimal alpha-Decision Trees , 2010, RSCTC.

[22]  Bernard M. E. Moret,et al.  The Activity of a Variable and Its Relation to Decision Trees , 1980, TOPL.

[23]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[24]  Igor Chikalov,et al.  A Tool for Study of Optimal Decision Trees , 2010, RSKT.

[25]  Roman Słowiński,et al.  Intelligent Decision Support , 1992, Theory and Decision Library.

[26]  Andrzej Skowron,et al.  The Discernibility Matrices and Functions in Information Systems , 1992, Intelligent Decision Support.

[27]  J. Kent Martin,et al.  An Exact Probability Metric for Decision Tree Splitting and Stopping , 1997, Machine Learning.

[28]  Alberto Martelli,et al.  Optimizing decision trees through heuristically guided search , 1978, CACM.

[29]  Xinhua Zhuang,et al.  Binary linear decision tree with genetic algorithm , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[30]  John Mingers,et al.  Expert Systems—Rule Induction with Statistical Data , 1987 .

[31]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[32]  Andrzej Skowron,et al.  Transactions on Rough Sets III , 2005, Trans. Rough Sets.

[33]  Yuri Breitbart,et al.  A branch-and-bound algorithm to obtain an optimal evaluation tree for monotonic Boolean functions , 2004, Acta Informatica.