Adaptive Model Rules From High-Speed Data Streams

Decision rules are one of the most expressive and interpretable models for machine learning. In this article, we present Adaptive Model Rules (AMRules), the first stream rule learning algorithm for regression problems. In AMRules, the antecedent of a rule is a conjunction of conditions on the attribute values, and the consequent is a linear combination of the attributes. In order to maintain a regression model compatible with the most recent state of the process generating data, each rule uses a Page-Hinkley test to detect changes in this process and react to changes by pruning the rule set. Online learning might be strongly affected by outliers. AMRules is also equipped with outliers detection mechanisms to avoid model adaption using anomalous examples. In the experimental section, we report the results of AMRules on benchmark regression problems, and compare the performance of our system with other streaming regression algorithms.

[1]  Geoff Holmes,et al.  MOA: Massive Online Analysis , 2010, J. Mach. Learn. Res..

[2]  Jiawei Han,et al.  ACM Transactions on Knowledge Discovery from Data: Introduction , 2007 .

[3]  Alípio Mário Jorge,et al.  Ensemble approaches for regression: A survey , 2012, CSUR.

[4]  Pedro M. Domingos.  A Unified Bias-Variance Decomposition , 2022 .

[5]  金田 重郎,et al.  C4.5: Programs for Machine Learning (書評) , 1995 .

[6]  Johannes Fürnkranz,et al.  Foundations of Rule Learning , 2012, Cognitive Technologies.

[7]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[8]  M WeissSholom,et al.  Rule-based machine learning methods for functional prediction , 1995 .

[9]  Raghu Ramakrishnan,et al.  Proceedings : KDD 2000 : the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 20-23, 2000, Boston, MA, USA , 2000 .

[10]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[11]  H. Mouss,et al.  Test of Page-Hinckley, an approach for fault detection in an agro-alimentary production system , 2004, 2004 5th Asian Control Conference (IEEE Cat. No.04EX904).

[12]  Geoff Hulten,et al.  Mining high-speed data streams , 2000, KDD '00.

[13]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[14]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[15]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[16]  J. R. Quinlan.  Learning With Continuous Classes , 1992 .

[17]  João Gama,et al.  Very Fast Decision Rules for multi-class problems , 2012, SAC '12.

[18]  BifetAlbert,et al.  MOA: Massive Online Analysis , 2010 .

[19]  J. Ross Quinlan,et al.  Combining Instance-Based and Model-Based Learning , 1993, ICML.

[20]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[21]  João Gama,et al.  On evaluating stream learning algorithms , 2012, Machine Learning.

[22]  Eyke Hüllermeier,et al.  IBLStreams: a system for instance-based classification and regression on data streams , 2012, Evol. Syst..

[23]  Philip S. Yu,et al.  Mining Concept-Drifting Data Streams , 2010, Data Mining and Knowledge Discovery Handbook.

[24]  Jesús S. Aguilar-Ruiz,et al.  Incremental Rule Learning and Border Examples Selection from Numerical Data Streams , 2005, J. Univers. Comput. Sci..

[25]  Yong Wang,et al.  Using Model Trees for Classification , 1998, Machine Learning.

[26]  C. Willmott,et al.  Advantages of the mean absolute error (MAE) over the root mean square error (RMSE) in assessing average model performance , 2005 .

[27]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[28]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[29]  Duncan Potts,et al.  Incremental learning of linear model trees , 2004, ICML.

[30]  B. B. Bhattacharyya.  One sided chebyshev inequality when the first four moments are known , 1987 .

[31]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[32]  João Gama,et al.  Adaptive Model Rules from Data Streams , 2013, ECML/PKDD.

[33]  Pedro M. Domingos.  A Unifeid Bias-Variance Decomposition and its Applications , 2000, ICML.

[34]  Volker Tresp,et al.  Robust Neural Network Regression for Offline and Online Learning , 1999, NIPS.

[35]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[36]  Sholom M. Weiss,et al.  Rule-based Machine Learning Methods for Functional Prediction , 1995, J. Artif. Intell. Res..

[37]  Andy Liaw,et al.  Classification and Regression by randomForest , 2007 .

[38]  João Gama,et al.  Random rules from data streams , 2013, SAC '13.

[39]  Haytham Elghazel,et al.  An Empirical Comparison of Supervised Ensemble Learning Approaches , 2013 .

[40]  Jesús S. Aguilar-Ruiz,et al.  Knowledge discovery from data streams , 2009, Intell. Data Anal..

[41]  Stuart J. Russell,et al.  Online bagging and boosting , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[42]  Saso Dzeroski,et al.  Learning model trees from evolving data streams , 2010, Data Mining and Knowledge Discovery.

[43]  Thomas G. Dietterich.  Multiple Classifier Systems , 2000, Lecture Notes in Computer Science.

[44]  Philip S. Yu,et al.  Mining concept-drifting data streams using ensemble classifiers , 2003, KDD '03.

[45]  M. I. Jordan.  Leo Breiman , 2011, 1101.0929.

[46]  Charles Yount,et al.  Using Model Trees for Computer Architecture Performance Analysis of Software Applications , 2007, 2007 IEEE International Symposium on Performance Analysis of Systems & Software.

[47]  Vipin Kumar,et al.  Chapman & Hall/CRC Data Mining and Knowledge Discovery Series , 2008 .

[48]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .