Pareto-optimality of oblique decision trees from evolutionary algorithms

This paper investigates the performance of evolutionary algorithms in the optimization aspects of oblique decision tree construction and describes their performance with respect to classification accuracy, tree size, and Pareto-optimality of their solution sets. The performance of the evolutionary algorithms is analyzed and compared to the performance of exhaustive (traditional) decision tree classifiers on several benchmark datasets. The results show that the classification accuracy and tree sizes generated by the evolutionary algorithms are comparable with the results generated by traditional methods in all the sample datasets and in the large datasets, the multiobjective evolutionary algorithms generate better Pareto-optimal sets than the sets generated by the exhaustive methods. The results also show that a classifier, whether exhaustive or evolutionary, that generates the most accurate trees does not necessarily generate the shortest trees or the best Pareto-optimal sets.

[1]  Giandomenico Spezzano,et al.  Genetic Programming and Simulated Annealing: A Hybrid Method to Evolve Decision Trees , 2000, EuroGP.

[2]  P. Pardalos,et al.  Pareto optimality, game theory and equilibria , 2008 .

[3]  Steven Salzberg,et al.  Lookahead and Pathology in Decision Tree Induction , 1995, IJCAI.

[4]  W. Loh,et al.  Tree-Structured Classification via Generalized Discriminant Analysis. , 1988 .

[5]  William B. Langdon,et al.  Application of Genetic Programming to Induction of Linear Classification Trees , 2000, EuroGP.

[6]  Kweku-Muata Osei-Bryson,et al.  Evaluation of decision trees: a multi-criteria approach , 2004, Comput. Oper. Res..

[7]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[8]  Kalyanmoy Deb,et al.  Multi-objective optimization using evolutionary algorithms , 2001, Wiley-Interscience series in systems and optimization.

[9]  Lothar Thiele,et al.  Multiobjective Optimization Using Evolutionary Algorithms - A Comparative Case Study , 1998, PPSN.

[10]  Chandrika Kamath,et al.  Inducing oblique decision trees with evolutionary algorithms , 2003, IEEE Trans. Evol. Comput..

[11]  Dimitrios Kalles,et al.  Breeding Decision Trees Using Evolutionary Techniques , 2001, ICML.

[12]  Bernard M. E. Moret,et al.  Decision Trees and Diagrams , 1982, CSUR.

[13]  Simon Kasif,et al.  Induction of Oblique Decision Trees , 1993, IJCAI.

[14]  Sunil Vadera,et al.  Inducing safer oblique trees without costs , 2005, Expert Syst. J. Knowl. Eng..

[15]  Panos M. Pardalos,et al.  A survey of recent developments in multiobjective optimization , 2007, Ann. Oper. Res..

[16]  Edward J. Delp,et al.  An iterative growing and pruning algorithm for classification tree design , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[17]  Chris Carter,et al.  Multiple decision trees , 2013, UAI.

[18]  W. Loh,et al.  SPLIT SELECTION METHODS FOR CLASSIFICATION TREES , 1997 .

[19]  Gerrit K. Janssens,et al.  Data mining with genetic algorithms on binary trees , 2003, Eur. J. Oper. Res..

[20]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[21]  John R. Koza,et al.  Hierarchical Genetic Algorithms Operating on Populations of Computer Programs , 1989, IJCAI.

[22]  Marco Laumanns,et al.  SPEA2: Improving the Strength Pareto Evolutionary Algorithm For Multiobjective Optimization , 2002 .

[23]  Bart Kuijpers,et al.  Simulated annealing in the construction of near-optimal decision trees , 1994 .

[24]  Sreerama K. Murthy,et al.  Automatic Construction of Decision Trees from Data: A Multi-Disciplinary Survey , 1998, Data Mining and Knowledge Discovery.

[25]  David E. Goldberg,et al.  Genetic Algorithms, Selection Schemes, and the Varying Effects of Noise , 1996, Evolutionary Computation.

[26]  Gary B. Lamont,et al.  Multiobjective Evolutionary Algorithms: Analyzing the State-of-the-Art , 2000, Evolutionary Computation.

[27]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[28]  Lothar Thiele,et al.  Comparison of Multiobjective Evolutionary Algorithms: Empirical Results , 2000, Evolutionary Computation.

[29]  Usama M. Fayyad,et al.  What Should Be Minimized in a Decision Tree? , 1990, AAAI.

[30]  Ching Y. Suen,et al.  Analysis and Design of a Decision Tree Based on Entropy Reduction and Its Application to Large Character Set Recognition , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Wray L. Buntine,et al.  Learning classification trees , 1992 .

[32]  Seymour Shlien.  Nonparametric classification using matched binary decision trees , 1992, Pattern Recognit. Lett..

[33]  Maria Dolores Gil Montoya,et al.  A hybrid method for solving multi-objective global optimization problems , 2007, J. Glob. Optim..

[34]  William Nick Street.  Oblique Multicategory Decision Trees Using Nonlinear Programming , 2005, INFORMS J. Comput..

[35]  Simon Kasif,et al.  A System for Induction of Oblique Decision Trees , 1994, J. Artif. Intell. Res..

[36]  David E. Goldberg,et al.  The Gambler's Ruin Problem, Genetic Algorithms, and the Sizing of Populations , 1999, Evolutionary Computation.

[37]  Constantin Zopounidis,et al.  An Experimental Evaluation of Some Classification Methods , 2006, J. Glob. Optim..

[38]  I. Bratko,et al.  Information-based evaluation criterion for classifier's performance , 2004, Machine Learning.

[39]  DaeEun Kim,et al.  Structural Risk Minimization on Decision Trees Using an Evolutionary Multiobjective Optimization , 2004, EuroGP.

[40]  Massimo Pappalardo,et al.  Multiobjective Optimization: A Brief Overview , 2008 .

[41]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[42]  Huimin Zhao,et al.  A multi-objective genetic programming approach to developing Pareto optimal decision trees , 2007, Decis. Support Syst..