Uncorrelated trace ratio linear discriminant analysis for undersampled problems

For linear discriminant analysis (LDA), the ratio trace and trace ratio are two basic criteria generalized from the classical Fisher criterion function, while the orthogonal and uncorrelated constraints are two common conditions imposed on the optimal linear transformation. The ratio trace criterion with both the orthogonal and uncorrelated constraints have been extensively studied in the literature, whereas the trace ratio criterion receives less interest mainly due to the lack of a closed-form solution and efficient algorithms. In this paper, we make an extensive study on the uncorrelated trace ratio linear discriminant analysis, with particular emphasis on the application on the undersampled problem. Two regularization uncorrelated trace ratio LDA models are discussed for which the global solutions are characterized and efficient algorithms are established. Experimental comparison on several LDA approaches are conducted on several real world datasets, and the results show that the uncorrelated trace ratio LDA is competitive with the orthogonal trace ratio LDA, but is better than the results based on ratio trace criteria in terms of the classification performance.

[1]  Marcel Dettling,et al.  BagBoosting for tumor classification with gene expression data , 2004, Bioinform..

[2]  David J. Kriegman,et al.  From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[4]  Michael K. Ng,et al.  Fast Algorithms for the Generalized Foley-Sammon Discriminant Analysis , 2010, SIAM J. Matrix Anal. Appl..

[5]  Aleix M. Martínez,et al.  Where are linear feature extraction methods applicable? , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Haesun Park,et al.  Fast Linear Discriminant Analysis using QR Decomposition and Regularization , 2007 .

[7]  Haesun Park,et al.  Equivalence of Several Two-Stage Methods for Linear Discriminant Analysis , 2004, SDM.

[8]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[9]  M. Ringnér,et al.  Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks , 2001, Nature Medicine.

[10]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[11]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[12]  Jieping Ye,et al.  Feature Reduction via Generalized Uncorrelated Linear Discriminant Analysis , 2006, IEEE Transactions on Knowledge and Data Engineering.

[13]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[14]  David G. Stork,et al.  Pattern Classification , 1973 .

[15]  Haesun Park,et al.  Generalizing discriminant analysis using the generalized singular value decomposition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Dong Xu,et al.  Trace Ratio vs. Ratio Trace for Dimensionality Reduction , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Feiping Nie,et al.  Semi-supervised orthogonal discriminant analysis via label propagation , 2009, Pattern Recognit..

[18]  Avinash C. Kak,et al.  PCA versus LDA , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Haesun Park,et al.  Structure Preserving Dimension Reduction for Clustered Text Data Based on the Generalized Singular Value Decomposition , 2003, SIAM J. Matrix Anal. Appl..

[20]  Jieping Ye,et al.  Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis , 2006, J. Mach. Learn. Res..

[21]  Shuicheng Yan,et al.  Graph embedding: a general framework for dimensionality reduction , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[23]  D. Ruppert.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2004 .

[24]  T. Poggio,et al.  Prediction of central nervous system embryonal tumour outcome based on gene expression , 2002, Nature.

[25]  Jing-Yu Yang,et al.  Face recognition based on the uncorrelated discriminant transformation , 2001, Pattern Recognit..

[26]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[27]  L. Duchene,et al.  An Optimal Transformation for Discriminant and Principal Component Analysis , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Xiaoyang Tan,et al.  A study on three linear discriminant analysis based methods in small sample size problem , 2008, Pattern Recognit..

[29]  Jing-Yu Yang,et al.  A generalized Foley-Sammon transform based on generalized fisher discriminant criterion and its application to face recognition , 2003, Pattern Recognit. Lett..

[30]  J. Mesirov,et al.  Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. , 1999, Science.

[31]  Juyang Weng,et al.  Using Discriminant Eigenfeatures for Image Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[32]  Jieping Ye,et al.  Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems , 2005, J. Mach. Learn. Res..

[33]  Jieping Ye,et al.  Efficient model selection for regularized linear discriminant analysis , 2006, CIKM '06.

[34]  W. V. McCarthy,et al.  Discriminant Analysis with Singular Covariance Matrices: Methods and Applications to Spectroscopic Data , 1995 .

[35]  T. Mexia,et al.  Author ' s personal copy , 2009 .

[36]  Jieping Ye,et al.  An optimization criterion for generalized discriminant analysis on undersampled problems , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Jing-Yu Yang,et al.  A theorem on the uncorrelated optimal discriminant vectors , 2001, Pattern Recognit..

[38]  John W. Sammon,et al.  An Optimal Set of Discriminant Vectors , 1975, IEEE Transactions on Computers.

[39]  Mohammed Bellalij,et al.  The Trace Ratio Optimization Problem , 2012, SIAM Rev..

[40]  G. McLachlan.  Discriminant Analysis and Statistical Pattern Recognition , 1992 .