Globally Optimal Fuzzy Decision Trees for Classification and Regression

A fuzzy decision tree is constructed by allowing the possibility of partial membership of a point in the nodes that make up the tree structure. This extension of its expressive capabilities transforms the decision tree into a powerful functional approximant that incorporates features of connectionist methods, while remaining easily interpretable. Fuzzification is achieved by superimposing a fuzzy structure over the skeleton of a CART decision tree. A training rule for fuzzy trees, similar to backpropagation in neural networks, is designed. This rule corresponds to a global optimization algorithm that fixes the parameters of the fuzzy splits. The method developed for the automatic generation of fuzzy decision trees is applied to both classification and regression problems. In regression problems, it is seen that the continuity constraint imposed by the function representation of the fuzzy tree leads to substantial improvements in the quality of the regression and limits the tendency to overfitting. In classification, fuzzification provides a means of uncovering the structure of the probability distribution for the classification errors in attribute space. This allows the identification of regions for which the error rate of the tree is significantly lower than the average error rate, sometimes even below the Bayes misclassification rate.

[1]  Ishwar K. Sethi,et al.  Entropy nets: from decision trees to neural networks , 1990, Proc. IEEE.

[2]  Edward J. Delp,et al.  An iterative growing and pruning algorithm for classification tree design , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[3]  Donato Malerba,et al.  A Comparative Analysis of Methods for Pruning Decision Trees , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[4]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[5]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994, Neural Computation.

[6]  Theodosios Pavlidis,et al.  Fuzzy Decision Tree Algorithms , 1977, IEEE Transactions on Systems, Man, and Cybernetics.

[7]  Robert A. Jacobs,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[8]  Lotfi A. Zadeh,et al.  Fuzzy Sets , 1996, Inf. Control..

[9]  Cezary Z. Janikow,et al.  Fuzzy decision trees: issues and methods , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[10]  J. Ross Quinlan,et al.  Decision trees and decision-making , 1990, IEEE Trans. Syst. Man Cybern..

[11]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[12]  Oscar H. IBARm.  Information and Control , 1957, Nature.

[13]  F. A. Seiler,et al.  Numerical Recipes in C: The Art of Scientific Computing , 1989 .

[14]  Nikola Kasabov,et al.  Foundations Of Neural Networks, Fuzzy Systems, And Knowledge Engineering [Books in Brief] , 1996, IEEE Transactions on Neural Networks.

[15]  Ching Y. Suen,et al.  Large Tree Classifier with Heuristic Search and Global Training , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Wayne Ieee,et al.  Entropy Nets: From Decision Trees to Neural Networks , 1990 .

[17]  Jude W. Shavlik.  Learning by symbolic and neural methods , 1998 .

[18]  L. Zadeh.  Fuzzy sets and their application to pattern classification and clustering analysis , 1996 .

[19]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[20]  Lotfi A. Zadeh,et al.  Outline of a New Approach to the Analysis of Complex Systems and Decision Processes , 1973, IEEE Trans. Syst. Man Cybern..

[21]  P.E. Maher,et al.  Uncertain reasoning in an ID3 machine learning framework , 1993, [Proceedings 1993] Second IEEE International Conference on Fuzzy Systems.

[22]  Tarun Khanna,et al.  Foundations of neural networks , 1990 .

[23]  J. R. Quinlan.  DECISION TREES AS PROBABILISTIC CLASSIFIERS , 1987 .

[24]  Dimiter Driankov,et al.  Fuzzy model identification - selected approaches , 1997 .

[25]  Youngtae Park.  A comparison of neural net classifiers and linear tree classifiers: Their similarities and differences , 1994, Pattern Recognit..

[26]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[27]  J.-S.R. Jang,et al.  Structure determination in fuzzy modeling: a fuzzy CART approach , 1994, Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference.

[28]  Edward J. Delp,et al.  An Iterative Growing and Pruning Algorithm for Classification Tree Design , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Ishwar K. Sethi,et al.  Structure-driven induction of decision tree classifiers through neural learning , 1997, Pattern Recognit..

[30]  Abraham Kandel,et al.  Fuzzy techniques in pattern recognition , 1982 .

[31]  Vladimir Cherkassky,et al.  Comparison of adaptive methods for function estimation from samples , 1996, IEEE Trans. Neural Networks.

[32]  Wolfgang Doster,et al.  A decision theoretic approach to hierarchical classifier design , 1984, Pattern Recognit..

[33]  Vladimir Cherkassky,et al.  Statistical and neural network techniques for nonparametric regression , 1994 .

[34]  Ishwar K. Sethi.  Neural implementation of tree classifiers , 1995, IEEE Trans. Syst. Man Cybern..

[35]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[36]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[37]  W. Press,et al.  Numerical Recipes in Fortran: The Art of Scientific Computing.@@@Numerical Recipes in C: The Art of Scientific Computing. , 1994 .