Hybrid decision tree

In this paper, a hybrid learning approach named hybrid decision tree (HDT) is proposed. HDT simulates human reasoning by using symbolic learning to do qualitative analysis and using neural learning to do subsequent quantitative analysis. It generates the trunk of a binary HDT according to the binary information gain ratio criterion in an instance space defined by only original unordered attributes. If unordered attributes cannot further distinguish training examples falling into a leaf node whose diversity is beyond the diversity-threshold, then the node is marked as a dummy node. After all those dummy nodes are marked, a specific feedforward neural network named Fannc that is trained in an instance space defined by only original ordered attributes is exploited to accomplish the learning task. Moreover, this paper distinguishes three kinds of incremental learning tasks. Two incremental learning procedures designed for example-incremental learning with different storage requirements are provided, which enables HDT to deal gracefully with data sets where new data are frequently appended. Also a hypothesis-driven constructive induction mechanism is provided, which enables HDT to generate compact concept descriptions.

[1]  Carla E. Brodley,et al.  Linear Machine Decision Trees , 1991 .

[2]  Stuart L. Crawford.  Extensions to the CART Algorithm , 1989, Int. J. Man Mach. Stud..

[3]  Terence D. Sanger,et al.  A tree-structured adaptive network for function approximation in high-dimensional spaces , 1991, IEEE Trans. Neural Networks.

[4]  Paul E. Utgoff,et al.  Decision Tree Induction Based on Efficient Tree Restructuring , 1997, Machine Learning.

[5]  Jonathan J. Oliver.  Decision Graphs - An Extension of Decision Trees , 1993 .

[6]  Douglas H. Fisher,et al.  A Case Study of Incremental Concept Induction , 1986, AAAI.

[7]  David Mulvaney,et al.  On-line learning of fuzzy decision trees for global path planning , 1999 .

[8]  N. B. Karayiannis,et al.  CNeT: competitive neural trees for pattern classification , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[9]  John R. Anderson,et al.  MACHINE LEARNING An Artificial Intelligence Approach , 2009 .

[10]  John Mingers,et al.  An Empirical Comparison of Pruning Methods for Decision Tree Induction , 1989, Machine Learning.

[11]  João Gama,et al.  Oblique Linear Tree , 1997, IDA.

[12]  Ron Kohavi,et al.  Supervised and Unsupervised Discretization of Continuous Features , 1995, ICML.

[13]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[14]  Larry A. Rendell,et al.  Constructive Induction On Decision Trees , 1989, IJCAI.

[15]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[16]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[17]  Marvin Minsky,et al.  Logical Versus Analogical or Symbolic Versus Connectionist or Neat Versus Scruffy , 1991, AI Mag..

[18]  Eric E. Bloedorn,et al.  Multistrategy constructive induction , 1997 .

[19]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[20]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[21]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[22]  M. Kubat,et al.  Decision trees can initialize radial-basis function networks , 1998, IEEE Trans. Neural Networks.

[23]  Richard J. Mammone,et al.  Optimal pruning of neural tree networks for improved generalization , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[24]  David McSherry,et al.  Strategic induction of decision trees , 1999, Knowl. Based Syst..

[25]  Huan Liu,et al.  A connectionist approach to generating oblique decision trees , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[26]  Ryszard S. Michalski,et al.  Data-Driven Constructive Induction , 1998, IEEE Intell. Syst..

[27]  A. Jennings,et al.  Structurally adaptive self-organizing neural trees , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[28]  Ron Kohavi,et al.  Error-Based and Entropy-Based Discretization of Continuous Features , 1996, KDD.

[29]  Ronald A. Cole,et al.  A performance comparison of trained multilayer perceptrons and trained classification trees , 1990 .

[30]  Simon Kasif,et al.  A System for Induction of Oblique Decision Trees , 1994, J. Artif. Intell. Res..

[31]  Jude W. Shavlik,et al.  in Advances in Neural Information Processing , 1996 .

[32]  Paul E. Utgoff,et al.  Incremental Induction of Decision Trees , 1989, Machine Learning.

[33]  Ishwar K. Sethi,et al.  Structure-driven induction of decision tree classifiers through neural learning , 1997, Pattern Recognit..

[34]  Robert Givan,et al.  Online Ensemble Learning: An Empirical Study , 2000, Machine Learning.

[35]  Marcus R. Frean,et al.  Small nets and short paths : optimising neural computation , 1990 .

[36]  Ryszard S. Michalski,et al.  A theory and methodology of inductive learning , 1993 .

[37]  Zijian Zheng,et al.  Constructing conjunctions using systematic search on decision trees , 1998, Knowl. Based Syst..

[38]  Guillaume Deffuant.  Neural units recruitment algorithm for generation of decision trees , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[39]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994, Neural Computation.

[40]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[41]  James A. Hendler,et al.  Developing Hybrid Symbolic/Connectionist Models , 1991 .

[42]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[43]  Alberto Suárez,et al.  Globally Optimal Fuzzy Decision Trees for Classification and Regression , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[44]  Carla E. Brodley,et al.  An Incremental Method for Finding Multivariate Splits for Decision Trees , 1990, ML.

[45]  Philip D. Wasserman,et al.  Advanced methods in neural computing , 1993, VNR computer library.

[46]  R. Krishnan,et al.  Extracting decision trees from trained neural networks , 1999, Pattern Recognit..

[47]  Wayne Ieee,et al.  Entropy Nets: From Decision Trees to Neural Networks , 1990 .

[48]  Zhi-Hua Zhou,et al.  FANNC: A Fast Adaptive Neural Network Classifier , 2000, Knowledge and Information Systems.

[49]  Paul E. Utgoff,et al.  Perceptron Trees : A Case Study in ybrid Concept epresentations , 1999 .

[50]  Robert A. Jacobs,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1993, Neural Computation.

[51]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[52]  Brian C. Lovell,et al.  The Multiscale Classifier , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[53]  Olcay Boz,et al.  Extracting decision trees from trained neural networks , 2002, KDD.

[54]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[55]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[56]  Dimitrios Kalles,et al.  Efficient Incremental Induction of Decision Trees , 1996, Machine Learning.

[57]  Saul B. Gelfand,et al.  Classification trees with neural network feature extraction , 1992, IEEE Trans. Neural Networks.

[58]  Bruno Apolloni,et al.  Learning fuzzy decision trees , 1998, Neural Networks.