A novel application of Hoeffding's inequality to decision trees construction for data streams

Decision trees are the commonly applied tools in the task of data stream classification. The most critical point in decision tree construction algorithm is the choice of the splitting attribute. In majority of algorithms existing in literature the splitting criterion is based on statistical bounds derived for split measure functions. In this paper we propose a totally new kind of splitting criterion. We derive statistical bounds for arguments of split measure function instead of deriving it for split measure function itself. This approach allows us to properly use the Hoeffding's inequality to obtain the required bounds. Based on this theoretical results we propose the Decision Trees based on the Fractions Approximation algorithm (DTFA). The algorithm exhibits satisfactory results of classification accuracy in numerical experiments. It is also compared with other existing in literature methods, demonstrating noticeably better performance.

[1]  Jing Liu,et al.  Ambiguous decision trees for mining concept-drifting data streams , 2009, Pattern Recognit. Lett..

[2]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[3]  Ludmila I. Kuncheva,et al.  PCA Feature Extraction for Change Detection in Multidimensional Unlabeled Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[4]  João Gama,et al.  Learning decision trees from dynamic data streams , 2005, SAC '05.

[5]  Geoff Holmes,et al.  Active Learning With Drifting Streaming Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[6]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[7]  Gerald Schaefer,et al.  An improved ensemble approach for imbalanced classification problems , 2013, 2013 IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI).

[8]  Cesare Alippi,et al.  Just-In-Time Classifiers for Recurrent Concepts , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Andreas S. Weigend,et al.  John A. Hertz, Anders S. Krogh, Richard G. Palmer, Introduction to the Theory of Neural Computation , 1993, Artif. Intell..

[10]  Ernestina Menasalvas Ruiz,et al.  Mining Recurring Concepts in a Dynamic Feature Space , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[11]  Philip S. Yu,et al.  Mining concept-drifting data streams using ensemble classifiers , 2003, KDD '03.

[12]  Ruoming Jin,et al.  Efficient decision tree construction on streaming data , 2003, KDD '03.

[13]  Alexey Tsymbal,et al.  The problem of concept drift: definitions and related work , 2004 .

[14]  GamaJoão,et al.  Decision trees for mining data streams , 2006 .

[15]  Mykola Pechenizkiy,et al.  Dealing With Concept Drifts in Process Mining , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Myra Spiliopoulou,et al.  Correcting the Usage of the Hoeffding Inequality in Stream Mining , 2013, IDA.

[17]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[18]  Geoff Hulten,et al.  Mining high-speed data streams , 2000, KDD '00.

[19]  Piotr Duda,et al.  Decision Trees for Mining Data Streams Based on the McDiarmid's Bound , 2013, IEEE Transactions on Knowledge and Data Engineering.

[20]  Robert Nowicki,et al.  A New Method of Improving Classification Accuracy of Decision Tree in Case of Incomplete Samples , 2013, ICAISC.

[21]  Charu C. Aggarwal,et al.  Data Streams - Models and Algorithms , 2014, Advances in Database Systems.

[22]  Geoff Holmes,et al.  New ensemble methods for evolving data streams , 2009, KDD.

[23]  Plamen P. Angelov,et al.  PANFIS: A Novel Incremental Learning Machine , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[24]  Jerzy Stefanowski,et al.  Reacting to Different Types of Concept Drift: The Accuracy Updated Ensemble Algorithm , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Geoff Hulten,et al.  Mining time-changing data streams , 2001, KDD '01.

[26]  Geoff Holmes,et al.  New Options for Hoeffding Trees , 2007, Australian Conference on Artificial Intelligence.

[27]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[28]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[29]  Albert Bifet,et al.  Adaptive Stream Mining: Pattern Learning and Mining from Evolving Data Streams , 2010, Frontiers in Artificial Intelligence and Applications.

[30]  J. Ross Quinlan,et al.  Learning Efficient Classification Procedures and Their Application to Chess End Games , 1983 .

[31]  Yu Fu,et al.  Soft-CsGDT: soft cost-sensitive Gaussian decision tree for cost-sensitive classification of data streams , 2013, BigMine '13.

[32]  João Gama,et al.  Decision trees for mining data streams , 2006, Intell. Data Anal..

[33]  Piotr Duda,et al.  Decision Trees for Mining Data Streams Based on the Gaussian Approximation , 2014, IEEE Transactions on Knowledge and Data Engineering.

[34]  Piotr Duda,et al.  The CART decision tree for mining data streams , 2014, Inf. Sci..