An Incremental Feature Learning Algorithm Based on Least Square Support Vector Machine

Incremental learning has been widely addressed in machine learning literature to deal with tasks where the learning environment is steadily changing or training samples become available one after another over time. Support Vector Machine has been successfully used in pattern recognition and function estimation. In order to tackle with incremental learning problems with new features, an incremental feature learning algorithm based on Least Square Support Vector Machine is proposed in this paper. In this algorithm, features of newly joined samples contain two parts: already existing features and new features. Using historic structural parameters which are trained from the already existing features, the algorithm only trains the new features with Least Square Support Vector Machine. Experiments show that this algorithm has two outstanding properties. First, different kernel functions can be used for the already existing features and the new features according to the distribution of samples. Consequently, this algorithm is more suitable to deal with classification tasks which can not be well solved by using a single kernel function. Second, the training time and the memory space can be reduced because the algorithm fully uses the structural parameters of classifiers trained formerly and only trains the new features with Least Square Support Vector Machine. Some UCI datasets are used to demonstrate the less training time and comparable or better performance of this algorithm than the Least Square Support Vector Machine.

[1]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[2]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[3]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[4]  Qinming He,et al.  An Incremental Learning Algorithm Based on Support Vector Domain Classifier , 2006, 2006 5th IEEE International Conference on Cognitive Informatics.

[5]  Nello Cristianini,et al.  An introduction to Support Vector Machines , 2000 .

[6]  Dimitrios Gunopulos,et al.  Incremental support vector machine construction , 2001, Proceedings 2001 IEEE International Conference on Data Mining.

[7]  Marimuthu Palaniswami,et al.  Incremental training of support vector machines , 2005, IEEE Transactions on Neural Networks.

[8]  B. De Moor,et al.  Ordinal Least Squares Support Vector Machines - A Discriminant Analysis Approach , 2006, 2006 16th IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing.

[10]  Gunnar Rätsch,et al.  Input space versus feature space in kernel-based methods , 1999, IEEE Trans. Neural Networks.

[11]  Alexander J. Smola,et al.  Learning the Kernel with Hyperkernels , 2005, J. Mach. Learn. Res..

[12]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[13]  Kazushi Ikeda,et al.  Incremental support vector machines and their geometrical analyses , 2007, Neurocomputing.