Incremental linear discriminant analysis for evolving feature spaces in multitask pattern recognition problems

In this paper, we propose a new incremental linear discriminant analysis (ILDA) for multitask pattern recognition (MTPR) problems in which a chunk of training data for a particular task are given sequentially and the task is switched to another related task one after another. The Pang et al.’s ILDA is extended such that a discriminant space of the current task is augmented with effective discriminant vectors that are selected from other tasks based on the class separability. We call this selective augmentation of discriminant vectors knowledge transfer of feature space. In the experiments, the proposed ILDA is evaluated for seven MTPR problems, each of which consists of three recognition tasks. The results demonstrate that the proposed ILDA with knowledge transfer outperforms the conventional ILDA and its naive extension to MTPR problems with regard to both class separability and recognition accuracy. We confirm that the proposed knowledge transfer works well to evolve effective feature spaces online in MTPR problems.

[1]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[2]  Shigeo Abe,et al.  Incremental learning of feature space and classifier for face recognition , 2005, Neural Networks.

[3]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[4]  Thomas Martinetz,et al.  SoftDoubleMaxMinOver: Perceptron-Like Training of Support Vector Machines , 2009, IEEE Transactions on Neural Networks.

[5]  Rich Caruana,et al.  Learning Many Related Tasks at the Same Time with Backpropagation , 1994, NIPS.

[6]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[7]  Massimiliano Pontil,et al.  Multi-Task Feature Learning , 2006, NIPS.

[8]  Shaoning Pang,et al.  Incremental Learning of Chunk Data for Online Pattern Classification Systems , 2008, IEEE Transactions on Neural Networks.

[9]  Shaoning Pang,et al.  Incremental linear discriminant analysis for classification of data streams , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[10]  Shingo Tomita,et al.  An optimal orthonormal system for discriminant analysis , 1985, Pattern Recognit..

[11]  Tong Heng Lee,et al.  Face recognition using recursive Fisher linear discriminant , 2006, IEEE Transactions on Image Processing.

[12]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[13]  Nikola K. Kasabov,et al.  Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning , 2001, IEEE Trans. Syst. Man Cybern. Part B.

[14]  Wenming Zheng.  Class-Incremental Generalized Discriminant Analysis , 2006, Neural Computation.

[15]  Seiichi Ozawa,et al.  A Multitask Learning Model for Online Pattern Recognition , 2009, IEEE Transactions on Neural Networks.

[16]  Shigeo Abe,et al.  An Efficient Incremental Kernel Principal Component Analysis for Online Feature Selection , 2007, 2007 International Joint Conference on Neural Networks.

[17]  Nikola Kasabov,et al.  Evolving Connectionist Systems: Methods and Applications in Bioinformatics, Brain Study and Intelligent Machines , 2002, IEEE Transactions on Neural Networks.

[18]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[19]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[20]  Yaser S. Abu-Mostafa,et al.  Learning from hints in neural networks , 1990, J. Complex..