Feature Extraction Using Sequential Semidefinite Programming

Many feature extraction approaches end up with a trace quotient formulation. Since it is difficult to directly solve the trace quotient problem, conventionally the trace quotient cost is replaced by an approximation such that the generalised eigen-decomposition can be applied. In this work we directly optimise the trace quotient. It is reformulated as a quasi-linear semidefinite optimisation problem, which can be solved globally and efficiently using standard off-the-shelf semidefinite programming solvers. Also this optimisation strategy allows one to enforce additional constraints ( e.g., sparseness constraints) on the projection matrix. Based on this optimisation framework, a novel feature extraction algorithm is designed. Its advantages are demonstrated on several UCI machine learning benchmark datasets, USPS handwritten digits and ORL face data.

[1]  Michael L. Overton,et al.  Optimality conditions and duality theory for minimizing sums of the largest eigenvalues of symmetric matrices , 2015, Math. Program..

[2]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[3]  Amir Globerson,et al.  Metric Learning by Collapsing Classes , 2005, NIPS.

[4]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[5]  Michael I. Jordan,et al.  Distance Metric Learning with Application to Clustering with Side-Information , 2002, NIPS.

[6]  Shuicheng Yan,et al.  Trace Quotient Problems Revisited , 2006, ECCV.

[7]  Nello Cristianini,et al.  Learning the Kernel Matrix with Semidefinite Programming , 2002, J. Mach. Learn. Res..

[8]  Daoqiang Zhang,et al.  Efficient and robust feature extraction by maximum margin criterion , 2003, IEEE Transactions on Neural Networks.

[9]  Michael L. Overton,et al.  On the Sum of the Largest Eigenvalues of a Symmetric Matrix , 1992, SIAM J. Matrix Anal. Appl..

[10]  Jos F. Sturm,et al.  A Matlab toolbox for optimization over symmetric cones , 1999 .

[11]  Jieping Ye,et al.  Null space versus orthogonal linear discriminant analysis , 2006, ICML '06.

[12]  Naftali Tishby,et al.  Margin based feature selection - theory and algorithms , 2004, ICML.

[13]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Arkadi Nemirovski,et al.  Lectures on modern convex optimization - analysis, algorithms, and engineering applications , 2001, MPS-SIAM series on optimization.

[15]  S. Schaible.  Fractional Programming. II, On Dinkelbach's Algorithm , 1976 .

[16]  B. Borchers.  CSDP, A C library for semidefinite programming , 1999 .

[17]  Kilian Q. Weinberger,et al.  Unsupervised Learning of Image Manifolds by Semidefinite Programming , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[18]  Patrik O. Hoyer,et al.  Non-negative Matrix Factorization with Sparseness Constraints , 2004, J. Mach. Learn. Res..

[19]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[20]  Luc Van Gool,et al.  SVM-based nonparametric discriminant analysis, an application to face detection , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[21]  Siegfried Schaible,et al.  Fractional Programming , 2009, Encyclopedia of Optimization.

[22]  Daniel Cremers,et al.  Binary Partitioning, Perceptual Grouping, and Restoration with Semidefinite Programming , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  B. Borchers.  A C library for semidefinite programming , 1999 .

[24]  Christoph Schnörr,et al.  Learning non-negative sparse image codes by convex programming , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[25]  Michael I. Jordan,et al.  A Direct Formulation for Sparse Pca Using Semidefinite Programming , 2004, SIAM Rev..

[26]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[27]  Christoph Schnörr,et al.  Controlling Sparseness in Non-negative Tensor Factorization , 2006, ECCV.