Ensemble learning for data stream analysis: A survey

A comprehensive survey of ensemble approaches for data stream analysis.Taxonomy of ensemble algorithms for various data stream mining tasks.Discussion of open research problems and lines of future research. In many applications of information systems learning algorithms have to act in dynamic environments where data are collected in the form of transient data streams. Compared to static data mining, processing streams imposes new computational requirements for algorithms to incrementally process incoming examples while using limited memory and time. Furthermore, due to the non-stationary characteristics of streaming data, prediction models are often also required to adapt to concept drifts. Out of several new proposed stream algorithms, ensembles play an important role, in particular for non-stationary environments. This paper surveys research on ensembles for data stream classification as well as regression tasks. Besides presenting a comprehensive spectrum of ensemble approaches for data streams, we also discuss advanced learning concepts such as imbalanced data streams, novelty detection, active and semi-supervised learning, complex data representations and structured outputs. The paper concludes with a discussion of open research problems and lines of future research.

[1]  J. Andel.  Sequential Analysis , 2022, The SAGE Encyclopedia of Research Design.

[2]  Mykola Pechenizkiy,et al.  An Overview of Concept Drift Applications , 2016 .

[3]  Xin Yao,et al.  Can cross-company data improve performance in software effort estimation? , 2012, PROMISE '12.

[4]  Sriganesh Madhvanath,et al.  Hybrid active learning for non-stationary streaming data with asynchronous labeling , 2015, 2015 IEEE International Conference on Big Data (Big Data).

[5]  Philip S. Yu,et al.  Mining concept-drifting data streams using ensemble classifiers , 2003, KDD '03.

[6]  Kyosuke Nishida,et al.  Adaptive Classifiers-Ensemble System for Tracking Concept Drift , 2007, 2007 International Conference on Machine Learning and Cybernetics.

[7]  Xin Yao,et al.  How to make best use of cross-company data in software effort estimation? , 2014, ICSE.

[8]  Robert Givan,et al.  Online Ensemble Learning: An Empirical Study , 2000, Machine Learning.

[9]  Michal Wozniak,et al.  Active Learning Classifier for Streaming Data , 2016, HAIS.

[10]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[11]  Luís Torgo,et al.  A Survey of Predictive Modelling under Imbalanced Distributions , 2015, ArXiv.

[12]  Jerzy Stefanowski,et al.  Prequential AUC: properties of the area under the ROC curve for data streams with concept drift , 2017, Knowledge and Information Systems.

[13]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[14]  Ludmila I. Kuncheva,et al.  Classifier Ensembles for Changing Environments , 2004, Multiple Classifier Systems.

[15]  Michal Wozniak,et al.  Concept Drift Detection and Model Selection with Simulated Recurrence and Ensembles of Statistical Detectors , 2013, J. Univers. Comput. Sci..

[16]  Thorsten Joachims,et al.  Detecting Concept Drift with Support Vector Machines , 2000, ICML.

[17]  Yee Whye Teh,et al.  Mondrian Forests: Efficient Online Random Forests , 2014, NIPS.

[18]  Ludmila I. Kuncheva,et al.  Classifier Ensembles for Detecting Concept Change in Streaming Data: Overview and Perspectives , 2008 .

[19]  Josef Raviv,et al.  Decision making in Markov chains applied to the problem of pattern recognition , 1967, IEEE Trans. Inf. Theory.

[20]  Myra Spiliopoulou,et al.  Framework for Storing and Processing Relational Entities in Stream Mining , 2013, PAKDD.

[21]  Cesare Alippi,et al.  Change detection tests using the ICI rule , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[22]  Robi Polikar,et al.  Incremental Learning of Concept Drift in Nonstationary Environments , 2011, IEEE Transactions on Neural Networks.

[23]  Charu C. Aggarwal,et al.  Classification and Adaptive Novel Class Detection of Feature-Evolving Data Streams , 2013, IEEE Transactions on Knowledge and Data Engineering.

[24]  Gerhard Widmer,et al.  Learning in the Presence of Concept Drift and Hidden Contexts , 1996, Machine Learning.

[25]  Francisco Herrera,et al.  Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power , 2010, Inf. Sci..

[26]  Roberto Souto Maior de Barros,et al.  A Lightweight Concept Drift Detection Ensemble , 2015, 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI).

[27]  Ralf Klinkenberg,et al.  Boosting classifiers for drifting concepts , 2007, Intell. Data Anal..

[28]  Kagan Tumer,et al.  Analysis of decision boundaries in linearly combined neural classifiers , 1996, Pattern Recognit..

[29]  Haibo He,et al.  Learning from Imbalanced Data , 2009, IEEE Transactions on Knowledge and Data Engineering.

[30]  Vasant Honavar,et al.  Learn++: an incremental learning algorithm for supervised neural networks , 2001, IEEE Trans. Syst. Man Cybern. Part C.

[31]  Albert Bifet,et al.  Sentiment Knowledge Discovery in Twitter Streaming Data , 2010, Discovery Science.

[32]  Kenneth O. Stanley.  Learning Concept Drift with a Committee of Decision Trees , 2003 .

[33]  Robi Polikar,et al.  Learn$^{++}$ .NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes , 2009, IEEE Transactions on Neural Networks.

[34]  Robi Polikar,et al.  COMPOSE: A Semisupervised Learning Framework for Initially Labeled Nonstationary Streaming Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[36]  Li Guo,et al.  Mining Multi-Label Data Streams Using Ensemble-Based Active Learning , 2012, SDM.

[37]  Dimitris K. Tasoulis,et al.  Exponentially weighted moving average charts for detecting concept drift , 2012, Pattern Recognit. Lett..

[38]  Ricard Gavaldà,et al.  Adaptive XML Tree Classification on Evolving Data Streams , 2009, ECML/PKDD.

[39]  Dan Roth,et al.  Learning cost-sensitive active classifiers , 2002, Artif. Intell..

[40]  Shigeo Abe,et al.  An Incremental Learning Algorithm of Ensemble Classifier Systems , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[41]  Katarzyna Musial,et al.  Next challenges for adaptive learning systems , 2012, SKDD.

[42]  Nathalie Japkowicz,et al.  Big Data Analysis: New Algorithms for a New Society , 2015 .

[43]  Robert M. Haralick,et al.  Decision Making in Context , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  Yunqian Ma,et al.  Imbalanced Learning: Foundations, Algorithms, and Applications , 2013 .

[45]  Konrad Jackowski,et al.  Fixed-size ensemble classifier system evolutionarily adapted to a recurring context with an unlimited pool of classifiers , 2013, Pattern Analysis and Applications.

[46]  Jerzy Stefanowski,et al.  Comparing Block Ensembles for Data Streams with Concept Drift , 2012, ADBIS Workshops.

[47]  Cesare Alippi.  Intelligence for Embedded Systems: A Methodological Approach , 2014 .

[48]  Marek Kurzynski,et al.  Multi-label Stream Classification Using Extended Binary Relevance Model , 2015, 2015 IEEE Trustcom/BigDataSE/ISPA.

[49]  Indre Zliobaite.  Controlled permutations for testing adaptive learning models , 2013, Knowledge and Information Systems.

[50]  Padraig Cunningham,et al.  A case-based technique for tracking concept drift in spam filtering , 2004, Knowl. Based Syst..

[51]  Ghazal Jaber,et al.  An approach for online learning in the presence of concept changes. (Une approche pour l'apprentissage en-ligne en présence de changements de concept.) , 2013 .

[52]  Xue Li,et al.  OcVFDT: one-class very fast decision tree for one-class classification of data streams , 2009, SensorKDD '09.

[53]  Anton Dries,et al.  Adaptive concept drift detection , 2009, SDM.

[54]  Padraig Cunningham,et al.  Using Diversity in Preparing Ensembles of Classifiers Based on Different Feature Subsets to Minimize Generalization Error , 2001, ECML.

[55]  Philip S. Yu,et al.  One-class learning and concept summarization for data streams , 2011, Knowledge and Information Systems.

[56]  Shan Juan Xie,et al.  Visual tracking with semi-supervised online weighted multiple instance learning , 2015, The Visual Computer.

[57]  Juan José Rodríguez Diez,et al.  Combining Online Classification Approaches for Changing Environments , 2008, SSPR/SPR.

[58]  Jerzy Stefanowski,et al.  Handling Sudden Concept Drift in Enron Messages Data Stream , 2010 .

[59]  Philip S. Yu,et al.  Pruning and dynamic scheduling of cost-sensitive ensembles , 2002, AAAI/IAAI.

[60]  Bogdan Gabrys,et al.  Local learning‐based adaptive soft sensor for catalyst activation prediction , 2011 .

[61]  Saso Dzeroski,et al.  Comparison of Tree-Based Methods for Multi-target Regression on Data Streams , 2015, NFMCP.

[62]  Saso Dzeroski,et al.  Online tree-based ensembles and option trees for regression on evolving data streams , 2015, Neurocomputing.

[63]  Eyke Hüllermeier,et al.  Recovery analysis for adaptive learning from non-stationary data streams: Experimental design and case study , 2015, Neurocomputing.

[64]  Roberto Souto Maior de Barros,et al.  A comparative study on concept drift detectors , 2014, Expert Syst. Appl..

[65]  Francisco Herrera,et al.  Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study , 2015, Knowledge and Information Systems.

[66]  Koichiro Yamauchi,et al.  Detecting Concept Drift Using Statistical Testing , 2007, Discovery Science.

[67]  Ricard Gavaldà,et al.  Learning from Time-Changing Data with Adaptive Windowing , 2007, SDM.

[68]  Indre Zliobaite,et al.  How good is the Electricity benchmark for evaluating concept drift adaptation , 2013, ArXiv.

[69]  Ralf Klinkenberg,et al.  Learning drifting concepts: Example selection vs. example weighting , 2004, Intell. Data Anal..

[70]  Mohak Shah,et al.  Evaluating Learning Algorithms: A Classification Perspective , 2011 .

[71]  Šarūnas Raudys,et al.  Statistical and Neural Classifiers: An Integrated Approach to Design , 2012 .

[72]  Stefan Rüping,et al.  Concept Drift and the Importance of Example , 2003, Text Mining.

[73]  Handbook of Parametric and Nonparametric Statistical Procedures , 2004 .

[74]  D. Kibler,et al.  Instance-based learning algorithms , 2004, Machine Learning.

[75]  Geoff Holmes,et al.  Active Learning with Evolving Streaming Data , 2011, ECML/PKDD.

[76]  Haibo He,et al.  Towards incremental learning of nonstationary imbalanced data stream: a multiple selectively recursive approach , 2011, Evol. Syst..

[77]  Gregory Ditzler,et al.  Learning in Nonstationary Environments: A Survey , 2015, IEEE Computational Intelligence Magazine.

[78]  Jerzy Stefanowski,et al.  Prequential AUC for Classifier Evaluation and Drift Detection in Evolving Data Streams , 2014, NFMCP.

[79]  David G. Stork,et al.  Pattern Classification , 1973 .

[80]  João Gama,et al.  Learning decision trees from dynamic data streams , 2005, SAC '05.

[81]  Michal Wozniak,et al.  A First Attempt on Online Data Stream Classifier Using Context , 2016, DMBD.

[82]  H. Hotelling.  The Generalization of Student’s Ratio , 1931 .

[83]  Bala Srinivasan,et al.  AnyNovel: detection of novel concepts in evolving data streams , 2016, Evol. Syst..

[84]  Nitesh V. Chawla,et al.  Learning in non-stationary environments with class imbalance , 2012, KDD.

[85]  Yolande Belaïd,et al.  An adaptive streaming active learning strategy based on instance weighting , 2016, Pattern Recognit. Lett..

[86]  Bhavani M. Thuraisingham,et al.  Classification and Novel Class Detection in Concept-Drifting Data Streams under Time Constraints , 2011, IEEE Transactions on Knowledge and Data Engineering.

[87]  Naoki Abe,et al.  Query Learning Strategies Using Boosting and Bagging , 1998, ICML.

[88]  Claudia Eckert,et al.  Lazy Gaussian Process Committee for Real-Time Online Regression , 2013, AAAI.

[89]  João Gama,et al.  A survey on concept drift adaptation , 2014, ACM Comput. Surv..

[90]  Xiaodong Lin,et al.  Active Learning From Stream Data Using Optimal Weight Classifier Ensemble , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[91]  Nitesh V. Chawla,et al.  Learning from streaming data with concept drift and imbalance: an overview , 2012, Progress in Artificial Intelligence.

[92]  Marcus A. Maloof,et al.  Using additive expert ensembles to cope with concept drift , 2005, ICML.

[93]  Nan Liu,et al.  Ensemble of subset online sequential extreme learning machine for class imbalance and concept drift , 2015, Neurocomputing.

[94]  Hamid Beigy,et al.  An ensemble of cluster-based classifiers for semi-supervised classification of non-stationary data streams , 2016, Knowledge and Information Systems.

[95]  J. Friedman,et al.  Multivariate generalizations of the Wald--Wolfowitz and Smirnov two-sample tests , 1979 .

[96]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[97]  Bartosz Krawczyk,et al.  Learning from imbalanced data: open challenges and future directions , 2016, Progress in Artificial Intelligence.

[98]  Wee Keong Ng,et al.  A survey on data stream clustering and classification , 2015, Knowledge and Information Systems.

[99]  Ralf Klinkenberg,et al.  An Ensemble Classifier for Drifting Concepts , 2005 .

[100]  Yang Zhang,et al.  Mining Multi-label Concept-Drifting Data Streams Using Dynamic Classifier Ensemble , 2009, ACML.

[101]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[102]  Geoff Holmes,et al.  Leveraging Bagging for Evolving Data Streams , 2010, ECML/PKDD.

[103]  Michal Wozniak,et al.  SCR: simulated concept recurrence – a non‐supervised tool for dealing with shifting concept , 2013, Expert Syst. J. Knowl. Eng..

[104]  David W. Aha,et al.  Instance-Based Learning Algorithms , 1991, Machine Learning.

[105]  Jesús S. Aguilar-Ruiz,et al.  Knowledge discovery from data streams , 2009, Intell. Data Anal..

[106]  André Carlos Ponce de Leon Ferreira de Carvalho,et al.  Evaluation of Multiclass Novelty Detection Algorithms for Data Streams , 2015, IEEE Transactions on Knowledge and Data Engineering.

[107]  Geoff Holmes,et al.  Improving Adaptive Bagging Methods for Evolving Data Streams , 2009, ACML.

[108]  Marcin Budka,et al.  Towards cost-sensitive adaptation: When is it worth updating your predictive model? , 2015, Neurocomputing.

[109]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[110]  Geoff Holmes,et al.  New ensemble methods for evolving data streams , 2009, KDD.

[111]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[112]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[113]  Lei Du,et al.  A Selective Detector Ensemble for Concept Drift Detection , 2015, Comput. J..

[114]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[115]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[116]  Grigorios Tsoumakas,et al.  An Ensemble of Classifiers for coping with Recurring Contexts in Data Streams , 2008, ECAI.

[117]  Bernard Zenko,et al.  Speeding-Up Hoeffding-Based Regression Trees With Options , 2011, ICML.

[118]  Xin Yao,et al.  Online Ensemble Learning of Data Streams with Gradually Evolved Classes , 2016, IEEE Transactions on Knowledge and Data Engineering.

[119]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[120]  Michal Wozniak,et al.  Active learning approach to concept drift problem , 2012, Log. J. IGPL.

[121]  Vipin Kumar,et al.  Chapman & Hall/CRC Data Mining and Knowledge Discovery Series , 2008 .

[122]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[123]  Bin Li,et al.  A survey on instance selection for active learning , 2012, Knowledge and Information Systems.

[124]  Cesare Alippi,et al.  Just-in-Time Adaptive Classifiers—Part I: Detecting Nonstationary Changes , 2008, IEEE Transactions on Neural Networks.

[125]  Misha Denil,et al.  Consistency of Online Random Forests , 2013, ICML.

[126]  Noel E. Sharkey,et al.  Combining diverse neural nets , 1997, The Knowledge Engineering Review.

[127]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[128]  Saso Dzeroski,et al.  Learning model trees from evolving data streams , 2010, Data Mining and Knowledge Discovery.

[129]  João Gama,et al.  Learning with Drift Detection , 2004, SBIA.

[130]  Xin Yao,et al.  The Impact of Diversity on Online Ensemble Learning in the Presence of Concept Drift , 2010, IEEE Transactions on Knowledge and Data Engineering.

[131]  Latifur Khan,et al.  Facing the reality of data stream classification: coping with scarcity of labeled data , 2012, Knowledge and Information Systems.

[132]  João Gama,et al.  Adaptive Model Rules From High-Speed Data Streams , 2014, BigMine.

[133]  Xin Yao,et al.  Resampling-Based Ensemble Methods for Online Class Imbalance Learning , 2015, IEEE Transactions on Knowledge and Data Engineering.

[134]  Dariusz Brzezinski,et al.  Structural XML Classification in Concept Drifting Data Streams , 2015, New Generation Computing.

[135]  Geoff Holmes,et al.  Accurate Ensembles for Data Streams: Combining Restricted Hoeffding Trees using Stacking , 2010, ACML.

[136]  M. R. Kmieciak,et al.  Semi-supervised approach to handle sudden concept drift in Enron data , 2011 .

[137]  Philip S. Yu,et al.  Uncertain One-Class Learning and Concept Summarization Learning on Uncertain Data Streams , 2014, IEEE Transactions on Knowledge and Data Engineering.

[138]  Geoff Holmes,et al.  New Options for Hoeffding Trees , 2007, Australian Conference on Artificial Intelligence.

[139]  Qiang-Li Zhao,et al.  Incremental Learning by Heterogeneous Bagging Ensemble , 2010, ADMA.

[140]  Geoff Holmes,et al.  Evaluation methods and decision theory for classification of streaming data with temporal dependence , 2015, Machine Learning.

[141]  Philip S. Yu,et al.  Graph stream classification using labeled and unlabeled graphs , 2013, 2013 IEEE 29th International Conference on Data Engineering (ICDE).

[142]  Piotr Jedrzejowicz,et al.  Ensemble Online Classifier Based on the One-Class Base Classifiers for Mining Data Streams , 2015, Cybern. Syst..

[143]  Magdalena Deckert.  Batch Weighted Ensemble for Mining Data Streams with Concept Drift , 2011, ISMIS.

[144]  Manfred K. Warmuth,et al.  THE WEIGHTED MAJORITY ALGORITHM (Supersedes 89-16) , 1992 .

[145]  Mohamed Medhat Gaber,et al.  A Survey of Classification Methods in Data Streams , 2007, Data Streams - Models and Algorithms.

[146]  João Gama,et al.  On evaluating stream learning algorithms , 2013, Machine Learning.

[147]  Cesare Alippi,et al.  Hierarchical Change-Detection Tests , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[148]  José del Campo-Ávila,et al.  Online and Non-Parametric Drift Detection Methods Based on Hoeffding’s Bounds , 2015, IEEE Transactions on Knowledge and Data Engineering.

[149]  Fredrik Gustafsson,et al.  Adaptive filtering and change detection , 2000 .

[150]  Eyke Hüllermeier,et al.  Survival analysis on data streams: Analyzing temporal events in dynamically changing environments , 2014, Int. J. Appl. Math. Comput. Sci..

[151]  Luís Torgo,et al.  A Survey of Predictive Modeling on Imbalanced Domains , 2016, ACM Comput. Surv..

[152]  Raj K. Bhatnagar,et al.  Tracking recurrent concept drift in streaming data using ensemble classifiers , 2007, ICMLA 2007.

[153]  Grigorios Tsoumakas,et al.  Dealing with Concept Drift and Class Imbalance in Multi-Label Stream Classification , 2011, IJCAI.

[154]  Gregory Ditzler,et al.  Incremental Learning of Concept Drift from Streaming Imbalanced Data , 2013, IEEE Transactions on Knowledge and Data Engineering.

[155]  D. Wolpert.  The Supervised Learning No-Free-Lunch Theorems , 2002 .

[156]  Carlo Zaniolo,et al.  Fast and Light Boosting for Adaptive Mining of Data Streams , 2004, PAKDD.

[157]  Chengqi Zhang,et al.  Graph Ensemble Boosting for Imbalanced Noisy Graph Stream Classification , 2015, IEEE Transactions on Cybernetics.

[158]  João Gama,et al.  A Study on Change Detection Methods , 2009 .

[159]  Alexey Tsymbal,et al.  The problem of concept drift: definitions and related work , 2004 .

[160]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[161]  Geoff Holmes,et al.  Scalable and efficient multi-label classification for evolving data streams , 2012, Machine Learning.

[162]  Haibo He,et al.  SERA: Selectively recursive approach towards nonstationary imbalanced stream data mining , 2009, 2009 International Joint Conference on Neural Networks.

[163]  Nitesh V. Chawla,et al.  Adaptive Methods for Classification in Arbitrarily Imbalanced and Drifting Data Streams , 2009, PAKDD Workshops.

[164]  Masayuki Takeda,et al.  Adaptive Online Prediction Using Weighted Windows , 2011, IEICE Trans. Inf. Syst..

[165]  Jerzy Stefanowski,et al.  Adaptive Ensembles for Evolving Data Streams - Combining Block-Based and Online Solutions , 2015, NFMCP.

[166]  Zhiquan Qi,et al.  Online multiple instance boosting for object detection , 2011, Neurocomputing.

[167]  Konrad Jackowski.  Adaptive Splitting and Selection Algorithm for Regression , 2015, New Generation Computing.

[168]  Xin Yao,et al.  Negative correlation in incremental learning , 2007, Natural Computing.

[169]  Jerzy Stefanowski,et al.  Reacting to Different Types of Concept Drift: The Accuracy Updated Ensemble Algorithm , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[170]  Philip S. Yu,et al.  Classifying Data Streams with Skewed Class Distributions and Concept Drifts , 2008, IEEE Internet Computing.

[171]  Xin Yao,et al.  Dealing with Multiple Classes in Online Class Imbalance Learning , 2016, IJCAI.

[172]  Philip S. Yu,et al.  Active Mining of Data Streams , 2004, SDM.

[173]  Michal Wozniak,et al.  Ensembles of Heterogeneous Concept Drift Detectors - Experimental Study , 2016, CISIM.

[174]  Kyosuke Nishida,et al.  Learning and Detecting Concept Drift , 2008 .

[175]  Bartosz Krawczyk,et al.  One-class classifiers with incremental learning and forgetting for data streams with concept drift , 2015, Soft Comput..

[176]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[177]  Fuad M. Alkoot.  Design of multiple classifier systems , 2001 .

[178]  Charu C. Aggarwal,et al.  Recurring and Novel Class Detection Using Class-Based Ensemble for Evolving Data Stream , 2016, IEEE Transactions on Knowledge and Data Engineering.

[179]  Albert Bifet,et al.  Adaptive learning and mining for data streams and frequent patterns , 2009, SKDD.

[180]  Geoffrey I. Webb,et al.  Characterizing concept drift , 2015, Data Mining and Knowledge Discovery.

[181]  Hadi Sadoghi Yazdi,et al.  Ensemble of online neural networks for non-stationary and imbalanced data streams , 2013, Neurocomputing.

[182]  H. Sebastian Seung,et al.  Selective Sampling Using the Query by Committee Algorithm , 1997, Machine Learning.

[183]  Jerzy Stefanowski,et al.  Ensemble Diversity in Evolving Data Streams , 2016, DS.

[184]  Horst Bischof,et al.  On-line Random Forests , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[185]  Geoff Hulten,et al.  Mining high-speed data streams , 2000, KDD '00.

[186]  Marcus A. Maloof,et al.  Dynamic weighted majority: a new ensemble method for tracking concept drift , 2003, Third IEEE International Conference on Data Mining.

[187]  Rui Araújo,et al.  An on-line weighted ensemble of regressor models to handle concept drifts , 2015, Eng. Appl. Artif. Intell..

[188]  Albert Bifet,et al.  Efficient Online Evaluation of Big Data Stream Classifiers , 2015, KDD.

[189]  Nikunj C. Oza,et al.  Online Ensemble Learning , 2000, AAAI/IAAI.

[190]  Eyke Hüllermeier,et al.  Open challenges for data stream mining research , 2014, SKDD.

[191]  Cesare Alippi,et al.  Intelligence for Embedded Systems , 2014 .

[192]  Geoff Holmes,et al.  CD-MOA: Change Detection Framework for Massive Online Analysis , 2013, IDA.

[193]  Michal Wozniak,et al.  Comparable Study of Statistical Tests for Virtual Concept Drift Detection , 2013, CORES.

[194]  Tin Kam Ho,et al.  Complexity of Classification Problems and Comparative Advantages of Combined Classifiers , 2000, Multiple Classifier Systems.

[195]  Bartosz Krawczyk,et al.  Incremental One-Class Bagging for Streaming and Evolving Big Data , 2015, 2015 IEEE Trustcom/BigDataSE/ISPA.

[196]  Jerzy Stefanowski,et al.  Accuracy Updated Ensemble for Data Streams with Concept Drift , 2011, HAIS.

[197]  Yuan Lan,et al.  Ensemble of online sequential extreme learning machine , 2009, Neurocomputing.

[198]  Yunming Ye,et al.  A new ensemble method for multi-label data stream classification in non-stationary environment , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[199]  William Nick Street,et al.  A streaming ensemble algorithm (SEA) for large-scale classification , 2001, KDD '01.

[200]  Georg Krempl,et al.  Classification in Presence of Drift and Latency , 2011, 2011 IEEE 11th International Conference on Data Mining Workshops.

[201]  Geoff Holmes,et al.  MOA: Massive Online Analysis , 2010, J. Mach. Learn. Res..

[202]  Jerzy Stefanowski,et al.  Classifiers for Concept-drifting Data Streams: Evaluating Things That Really Matter , 2013 .

[203]  Li Guo,et al.  Classifier and Cluster Ensembles for Mining Concept Drifting Data Streams , 2010, 2010 IEEE International Conference on Data Mining.

[204]  Rui Araújo,et al.  A dynamic and on-line ensemble regression for changing environments , 2015, Expert Syst. Appl..

[205]  A. Bifet,et al.  Early Drift Detection Method , 2005 .

[206]  Vincent Lemaire,et al.  A Survey on Supervised Classification on Data Streams , 2014, eBISS.

[207]  Michal Wozniak,et al.  Application of Combined Classifiers to Data Stream Classification , 2013, CISIM.

[208]  Michal Wozniak,et al.  Active Learning Classification of Drifted Streaming Data , 2016, ICCS.

[209]  Herna L. Viktor,et al.  Fast Hoeffding Drift Detection Method for Evolving Data Streams , 2016, ECML/PKDD.

[210]  Emilio Corchado,et al.  A survey of multiple classifier systems as hybrid systems , 2014, Inf. Fusion.

[211]  Thomas Seidl,et al.  Subspace clustering of data streams: new algorithms and effective evaluation measures , 2014, Journal of Intelligent Information Systems.

[212]  Xin Yao,et al.  DDD: A New Ensemble Approach for Dealing with Concept Drift , 2012, IEEE Transactions on Knowledge and Data Engineering.

[213]  Marcus A. Maloof,et al.  Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts , 2007, J. Mach. Learn. Res..

[214]  Herbert K. H. Lee,et al.  Lossless Online Bayesian Bagging , 2004, J. Mach. Learn. Res..

[215]  Jerzy Stefanowski,et al.  Combining block-based and online methods in learning ensembles from concept drifting data streams , 2014, Inf. Sci..

[216]  Geoff Holmes,et al.  Active Learning With Drifting Streaming Data , 2014, IEEE Transactions on Neural Networks and Learning Systems.