Analysis of correlation based dimension reduction methods

Analysis of correlation based dimension reduction methods Dimension reduction is an important topic in data mining and machine learning. Especially dimension reduction combined with feature fusion is an effective preprocessing step when the data are described by multiple feature sets. Canonical Correlation Analysis (CCA) and Discriminative Canonical Correlation Analysis (DCCA) are feature fusion methods based on correlation. However, they are different in that DCCA is a supervised method utilizing class label information, while CCA is an unsupervised method. It has been shown that the classification performance of DCCA is superior to that of CCA due to the discriminative power using class label information. On the other hand, Linear Discriminant Analysis (LDA) is a supervised dimension reduction method and it is known as a special case of CCA. In this paper, we analyze the relationship between DCCA and LDA, showing that the projective directions by DCCA are equal to the ones obtained from LDA with respect to an orthogonal transformation. Using the relation with LDA, we propose a new method that can enhance the performance of DCCA. The experimental results show that the proposed method exhibits better classification performance than the original DCCA.

[1]  Stephen A. Billings,et al.  Nonlinear Fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm , 2002, Neural Networks.

[2]  Jian Yang,et al.  Why can LDA be performed in PCA transformed space? , 2003, Pattern Recognit..

[3]  Jian Yang,et al.  Feature fusion: parallel strategy vs. serial strategy , 2003, Pattern Recognit..

[4]  P. Garthwaite.  An Interpretation of Partial Least Squares , 1994 .

[5]  Haesun Park,et al.  Generalizing discriminant analysis using the generalized singular value decomposition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Shuicheng Yan,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007 .

[7]  Haesun Park,et al.  A comparison of generalized linear discriminant analysis algorithms , 2008, Pattern Recognit..

[8]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[9]  Ja-Chen Lin,et al.  A new LDA-based face recognition system which can solve the small sample size problem , 1998, Pattern Recognit..

[10]  K. Fukunaga,et al.  Nonparametric Discriminant Analysis , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[12]  G. Baudat,et al.  Generalized Discriminant Analysis Using a Kernel Approach , 2000, Neural Computation.

[13]  Masashi Sugiyama,et al.  Local Fisher discriminant analysis for supervised dimensionality reduction , 2006, ICML.

[14]  Songcan Chen,et al.  Class label versus sample label-based CCA , 2007, Appl. Math. Comput..

[15]  Keinosuke Fukunaga,et al.  Introduction to statistical pattern recognition (2nd ed.) , 1990 .

[16]  H. Anton,et al.  Contemporary Linear Algebra , 2002 .

[17]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[18]  Yan Liu,et al.  A new method of feature fusion and its application in image recognition , 2005, Pattern Recognit..

[19]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[20]  Hua Yu,et al.  A direct LDA algorithm for high-dimensional data - with application to face recognition , 2001, Pattern Recognit..

[21]  Pengfei Shi,et al.  A Novel Method of Combined Feature Extraction for Recognition , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[22]  Songcan Chen,et al.  A Supervised Combined Feature Extraction Method for Recognition , 2008 .

[23]  Panos M. Pardalos,et al.  Data Mining and Mathematical Programming , 2008 .

[24]  I. Jolliffe.  Principal Component Analysis , 2002 .

[25]  Jacob A. Wegelin,et al.  A Survey of Partial Least Squares (PLS) Methods, with Emphasis on the Two-Block Case , 2000 .

[26]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Jieping Ye,et al.  Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems , 2005, J. Mach. Learn. Res..

[28]  Feiping Nie,et al.  Neighborhood MinMax Projections , 2007, IJCAI.

[29]  Pavel Pudil,et al.  Introduction to Statistical Pattern Recognition , 2006 .

[30]  Feiping Nie,et al.  Learning an Orthogonal and Smooth Subspace for Image Classification , 2009, IEEE Signal Processing Letters.

[31]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.