Incremental and Decremental SVM for Regression

Training a support vector machine (SVM) for regression (function approximation) in an incremental/decremental way consists essentially in migrating the input vectors in and out of the support vector set with specific modification of the associated thresholds. We introduce with full details such a method, which allows for defining the exact increments or decrements associated with the thresholds before vector migrations take place. Two delicate issues are especially addressed: the variation of the regularization parameter (for tuning the model performance) and the extreme situations where the support vector set becomes empty. We experimentally compare our method with several regression methods: the multilayer perceptron, two standard SVM implementations, and two models based on adaptive resonance theory.

[1]  Ichiro Takeuchi,et al.  Multiple Incremental Decremental Learning of Support Vector Machines , 2009, IEEE Transactions on Neural Networks.

[2]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[3]  Stephen Grossberg,et al.  Fuzzy ARTMAP: A neural network architecture for incremental supervised learning of analog multidimensional maps , 1992, IEEE Trans. Neural Networks.

[4]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[5]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[6]  Razvan Andonie,et al.  Bayesian ARTMAP for regression , 2013, Neural Networks.

[7]  M. Kubát.  An Introduction to Machine Learning , 2017, Springer International Publishing.

[8]  Boaz Lerner,et al.  The Bayesian ARTMAP , 2007, IEEE Transactions on Neural Networks.

[9]  Bin Gu,et al.  Accurate on-line v-support vector learning , 2012, Neural Networks.

[10]  Mario Martín Muñoz.  On-line support vector machines for function approximation , 2002 .

[11]  James Theiler,et al.  Accurate On-line Support Vector Regression , 2003, Neural Computation.

[12]  Gert Cauwenberghs,et al.  SVM incremental learning, adaptation and optimization , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[13]  Bernhard Schölkopf,et al.  New Support Vector Algorithms , 2000, Neural Computation.

[14]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[15]  Amnon Shashua,et al.  Introduction to Machine Learning: Class Notes 67577 , 2009, ArXiv.

[16]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[17]  Klaus-Robert Müller,et al.  Incremental Support Vector Learning: Analysis, Implementation and Applications , 2006, J. Mach. Learn. Res..

[18]  R. Andonie,et al.  Incremental / decremental SVM for function approximation , 2008, 2008 11th International Conference on Optimization of Electrical and Electronic Equipment.

[19]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[20]  Razvan Andonie,et al.  Fuzzy ARTMAP with input relevances , 2006, IEEE Transactions on Neural Networks.

[21]  Gene H. Golub,et al.  Matrix computations , 1983 .

[22]  Robert F. Harrison,et al.  A modified fuzzy ARTMAP architecture for the approximation of noisy mappings , 1995, Neural Networks.