How to Estimate the Regularization Parameter for Spectral Regression Discriminant Analysis and its Kernel Version?

Spectral regression discriminant analysis (SRDA) has recently been proposed as an efficient solution to large-scale subspace learning problems. There is a tunable regularization parameter in SRDA, which is critical to algorithm performance. However, how to automatically set this parameter has not been well solved until now. So this regularization parameter was only set to be a constant in SRDA, which is obviously suboptimal. This paper proposes to automatically estimate the optimal regularization parameter of SRDA based on the perturbation linear discriminant analysis (PLDA). In addition, two parameter estimation methods for the kernel version of SRDA are also developed. One is derived from the method of optimal regularization parameter estimation for SRDA. The other is to utilize the kernel version of PLDA. Experiments on a number of publicly available databases demonstrate the effectiveness of the proposed methods for face recognition, spoken letter recognition, handwritten digit recognition, and text categorization.

[1]  Lianwen Jin,et al.  Discriminative information preservation for face recognition , 2012, Neurocomputing.

[2]  Xuelong Li,et al.  Binary Two-Dimensional PCA , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[3]  Carsten Keßler,et al.  What is the difference? A cognitive dissimilarity measure for information retrieval result sets , 2012, Knowledge and Information Systems.

[4]  Dacheng Tao,et al.  Simple Exponential Family PCA , 2010, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Yun Fu,et al.  Image Classification Using Correlation Tensor Analysis , 2008, IEEE Transactions on Image Processing.

[6]  Dacheng Tao,et al.  Sparse transfer learning for interactive video search reranking , 2012, TOMCCAP.

[7]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[8]  Xiaogang Wang,et al.  A unified framework for subspace face recognition , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Wei Jia,et al.  Discriminant sparse neighborhood preserving embedding for face recognition , 2012, Pattern Recognit..

[10]  Jian Yang,et al.  BDPCA plus LDA: a novel fast feature extraction technique for face recognition , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[11]  Xuelong Li,et al.  Discriminant Locally Linear Embedding With High-Order Tensor Data , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[12]  Daoqiang Zhang,et al.  Semi-Supervised Dimensionality Reduction ∗ , 2007 .

[13]  Jian-Huang Lai,et al.  Perturbation LDA: Learning the difference between the class empirical mean and its expectation , 2009, Pattern Recognit..

[14]  Jian Yang,et al.  Constructing PCA Baseline Algorithms to Reevaluate ICA-Based Face-Recognition Performance , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[15]  Zhigang Luo,et al.  NeNMF: An Optimal Gradient Method for Nonnegative Matrix Factorization , 2012, IEEE Transactions on Signal Processing.

[16]  Stephen Lin,et al.  Rank-one Projections with Adaptive Margins for Face Recognition , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[17]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Zhigang Luo,et al.  Non-Negative Patch Alignment Framework , 2011, IEEE Transactions on Neural Networks.

[19]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[20]  Xuelong Li,et al.  Discriminative Orthogonal Neighborhood-Preserving Projections for Classification , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[21]  Shuicheng Yan,et al.  Ubiquitously Supervised Subspace Learning , 2009, IEEE Transactions on Image Processing.

[22]  Nizar Bouguila,et al.  A countably infinite mixture model for clustering and feature selection , 2011, Knowledge and Information Systems.

[23]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[24]  Jieping Ye,et al.  Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis , 2006, J. Mach. Learn. Res..

[25]  Tieniu Tan,et al.  Regularization parameter estimation for spectral regression discriminant analysis based on perturbation theory , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[26]  Xian-Sheng Hua,et al.  Ensemble Manifold Regularization , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Jin-Kao Hao,et al.  BicFinder: a biclustering algorithm for microarray data analysis , 2012, Knowledge and Information Systems.

[28]  Jieping Ye,et al.  Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection , 2008, IEEE Transactions on Neural Networks.

[29]  Stefan Axelsson,et al.  Similarity assessment for removal of noisy end user license agreements , 2011, Knowledge and Information Systems.

[30]  Yong Luo,et al.  Manifold Regularized Multitask Learning for Semi-Supervised Multilabel Image Classification , 2013, IEEE Transactions on Image Processing.

[31]  G. Baudat,et al.  Generalized Discriminant Analysis Using a Kernel Approach , 2000, Neural Computation.

[32]  Jie Gui,et al.  Multi-step dimensionality reduction and semi-supervised graph-based tumor classification using gene expression data , 2010, Artif. Intell. Medicine.

[33]  Mikhail Belkin,et al.  Semi-Supervised Learning on Riemannian Manifolds , 2004, Machine Learning.

[34]  Wei Jia,et al.  Locality preserving discriminant projections for face and palmprint recognition , 2010, Neurocomputing.

[35]  Jiawei Han,et al.  Semi-supervised Discriminant Analysis , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[36]  Jiawei Han,et al.  SRDA: An Efficient Algorithm for Large-Scale Discriminant Analysis , 2008, IEEE Transactions on Knowledge and Data Engineering.

[37]  Yan Gao,et al.  Similar Handwritten Chinese Character Recognition Using Discriminative Locality Alignment Manifold Learning , 2011, 2011 International Conference on Document Analysis and Recognition.

[38]  Daoqiang Zhang,et al.  Efficient and robust feature extraction by maximum margin criterion , 2003, IEEE Transactions on Neural Networks.

[39]  Soo-Young Lee,et al.  Extraction of independent discriminant features for data with asymmetric distribution , 2011, Knowledge and Information Systems.

[40]  Xuelong Li,et al.  Geometric Mean for Subspace Selection , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Jun Yu,et al.  On Combining Multiple Features for Cartoon Character Retrieval and Clip Synthesis , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[42]  Jiawei Han,et al.  Efficient Kernel Discriminant Analysis via Spectral Regression , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[43]  Xuelong Li,et al.  Semisupervised Dimensionality Reduction and Classification Through Virtual Label Regression , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[44]  Tao Jiang,et al.  Efficient and robust feature extraction by maximum margin criterion , 2003, IEEE Transactions on Neural Networks.

[45]  Shuicheng Yan,et al.  Correlation Metric for Generalized Feature Extraction , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Jonathan J. Hull,et al.  A Database for Handwritten Text Recognition Research , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  Dacheng Tao,et al.  Double Shrinking Sparse Dimension Reduction , 2013, IEEE Transactions on Image Processing.

[48]  Xuelong Li,et al.  General Tensor Discriminant Analysis and Gabor Features for Gait Recognition , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Lei Zhang,et al.  A multi-manifold discriminant analysis method for image feature extraction , 2011, Pattern Recognit..

[50]  Xuelong Li,et al.  Hessian Regularized Support Vector Machines for Mobile Image Annotation on the Cloud , 2013, IEEE Transactions on Multimedia.

[51]  Xian-Sheng Hua,et al.  Active Reranking for Web Image Search , 2010, IEEE Transactions on Image Processing.

[52]  Gerard de Haan,et al.  Optimal Regularization Parameter Estimation for Spectral Regression Discriminant Analysis , 2009, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Marie-Francine Moens,et al.  Highly discriminative statistical features for email classification , 2012, Knowledge and Information Systems.

[54]  Philip S. Yu,et al.  Under Consideration for Publication in Knowledge and Information Systems Gmlc: a Multi-label Feature Selection Framework for Graph Classification , 2011 .

[55]  Feiping Nie,et al.  A unified framework for semi-supervised dimensionality reduction , 2008, Pattern Recognit..

[56]  Dacheng Tao,et al.  Constrained Empirical Risk Minimization Framework for Distance Metric Learning , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[57]  Dacheng Tao,et al.  On Preserving Original Variables in Bayesian PCA With Application to Image Analysis , 2012, IEEE Transactions on Image Processing.

[58]  Jieping Ye,et al.  Least squares linear discriminant analysis , 2007, ICML '07.

[59]  Sanjay Chawla,et al.  Density-preserving projections for large-scale local anomaly detection , 2012, Knowledge and Information Systems.

[60]  Bo Geng,et al.  Manifold Regularized Multi-task Learning for Semi-supervised Multi-label Image Classification , 2013 .

[61]  John Shawe-Taylor,et al.  MahNMF: Manhattan Non-negative Matrix Factorization , 2012, ArXiv.

[62]  Meng Wang,et al.  Semisupervised Multiview Distance Metric Learning for Cartoon Synthesis , 2012, IEEE Transactions on Image Processing.

[63]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Cor J. Veenman,et al.  Visual Word Ambiguity , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Dacheng Tao,et al.  Subspaces Indexing Model on Grassmann Manifold for Image Search , 2011, IEEE Transactions on Image Processing.

[66]  Jiawei Han,et al.  Speed up kernel discriminant analysis , 2011, The VLDB Journal.