Feature Selective Projection with Low-Rank Embedding and Dual Laplacian Regularization

Feature extraction and feature selection have been regarded as two independent dimensionality reduction methods in most of the existing literature. In this paper, we propose to integrate both approaches into a unified framework and design an unsupervised linear feature selective projection (FSP) for feature extraction with low-rank embedding and dual Laplacian regularization, with the aim to exploit the intrinsic relationship among data and suppress the impact of noise. Specifically, a projection matrix with an <inline-formula><tex-math notation="LaTeX">$l_{2,1}$</tex-math><alternatives><mml:math><mml:msub><mml:mi>l</mml:mi><mml:mrow><mml:mn>2</mml:mn><mml:mo>,</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:msub></mml:math><inline-graphic xlink:href="liu-ieq1-2911946.gif"/></alternatives></inline-formula>-norm regularization is introduced to project original high dimensional data points into a new subspace with lower dimension, where the <inline-formula><tex-math notation="LaTeX">$l_{2,1}$</tex-math><alternatives><mml:math><mml:msub><mml:mi>l</mml:mi><mml:mrow><mml:mn>2</mml:mn><mml:mo>,</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:msub></mml:math><inline-graphic xlink:href="liu-ieq2-2911946.gif"/></alternatives></inline-formula>-norm regularization can endow the projection with good interpretability. We deploy a coefficient matrix with low rank constraint to reconstruct the data points and the <inline-formula><tex-math notation="LaTeX">$l_{2,1}$</tex-math><alternatives><mml:math><mml:msub><mml:mi>l</mml:mi><mml:mrow><mml:mn>2</mml:mn><mml:mo>,</mml:mo><mml:mn>1</mml:mn></mml:mrow></mml:msub></mml:math><inline-graphic xlink:href="liu-ieq3-2911946.gif"/></alternatives></inline-formula>-norm is imposed to regularize the data reconstruction errors in the low-dimensional subspace and make FSP robust to noise. Furthermore, a dual graph Laplacian regularization term is imposed on the low dimensional data and data reconstruction matrix for preserving the local manifold geometrical structure of data. Finally, an alternatively iterative algorithm is carefully designed for solving the proposed optimization model. Theoretical convergence and computational complexity analysis of the algorithm are also provided. Comprehensive experiments on various benchmark datasets have been carried out to evaluate the performance of the proposed FSP. As indicated, our algorithm significantly outperforms other state-of-the-art methods for feature extraction.

[1]  Yi Ma,et al.  Robust principal component analysis? , 2009, JACM.

[2]  Xiaofeng Zhu,et al.  Adaptive Hypergraph Learning for Unsupervised Feature Selection , 2017, IJCAI.

[3]  Ivor W. Tsang,et al.  Flexible Manifold Embedding: A Framework for Semi-Supervised and Unsupervised Dimension Reduction , 2010, IEEE Transactions on Image Processing.

[4]  Lin Wu,et al.  Robust Subspace Clustering for Multi-View Data by Exploiting Correlation Consensus , 2015, IEEE Transactions on Image Processing.

[5]  Thomas S. Huang,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation. , 2011, IEEE transactions on pattern analysis and machine intelligence.

[6]  Xiangtao Zheng,et al.  Discovering Diverse Subset for Unsupervised Hyperspectral Band Selection , 2017, IEEE Transactions on Image Processing.

[7]  Shuicheng Yan,et al.  Neighborhood preserving embedding , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[8]  Zhixun Su,et al.  Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation , 2011, NIPS.

[9]  C. A. Murthy.  Bridging Feature Selection and Extraction: Compound Feature Generation , 2017, IEEE Transactions on Knowledge and Data Engineering.

[10]  Qinghua Hu,et al.  Flexible Multi-View Dimensionality Co-Reduction , 2017, IEEE Transactions on Image Processing.

[11]  Jian Yang,et al.  BDPCA plus LDA: a novel fast feature extraction technique for face recognition , 2006, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[12]  George Atia,et al.  High Dimensional Low Rank Plus Sparse Matrix Decomposition , 2015, IEEE Transactions on Signal Processing.

[13]  Xuelong Li,et al.  Low-Rank Preserving Projections , 2016, IEEE Transactions on Cybernetics.

[14]  Yanwei Pang,et al.  Learning Regularized LDA by Clustering , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[15]  Hanqing Lu,et al.  Improving kernel Fisher discriminant analysis for face recognition , 2004, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Jane You,et al.  Robust Dual Clustering with Adaptive Manifold Regularization , 2017, IEEE Transactions on Knowledge and Data Engineering.

[17]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[18]  Huiqing Li,et al.  A Minimax Framework for Classification with Applications to Images and High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  I. Jolliffe.  Principal Component Analysis and Factor Analysis , 1986 .

[20]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[21]  Xiaoyang Tan,et al.  Pattern Recognition , 2016, Communications in Computer and Information Science.

[22]  Dacheng Tao,et al.  Double Shrinking Sparse Dimension Reduction , 2013, IEEE Transactions on Image Processing.

[23]  Zongben Xu,et al.  Enhancing Low-Rank Subspace Clustering by Manifold Regularization , 2014, IEEE Transactions on Image Processing.

[24]  Harry Wechsler,et al.  The FERET database and evaluation procedure for face-recognition algorithms , 1998, Image Vis. Comput..

[25]  Yong Luo,et al.  Tensor Canonical Correlation Analysis for Multi-View Dimension Reduction , 2015, IEEE Trans. Knowl. Data Eng..

[26]  Deng Cai,et al.  Unsupervised feature selection for multi-cluster data , 2010, KDD.

[27]  Huan Liu,et al.  Spectral feature selection for supervised and unsupervised learning , 2007, ICML '07.

[28]  Lin Wu,et al.  Iterative Views Agreement: An Iterative Low-Rank Based Structured Optimization Method to Multi-View Spectral Clustering , 2016, IJCAI.

[29]  Mohamed Nadif,et al.  A Semi-NMF-PCA Unified Framework for Data Clustering , 2017, IEEE Transactions on Knowledge and Data Engineering.

[30]  Xuelong Li,et al.  Unsupervised Feature Selection with Structured Graph Optimization , 2016, AAAI.

[31]  Feiping Nie,et al.  Orthogonal locality minimizing globality maximizing projections for feature extraction , 2009 .

[32]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[33]  Aleksandra Pizurica,et al.  Semisupervised Local Discriminant Analysis for Feature Extraction in Hyperspectral Images , 2013, IEEE Transactions on Geoscience and Remote Sensing.

[34]  Zhang Yi,et al.  Robust Subspace Clustering via Thresholding Ridge Regression , 2015, AAAI.

[35]  E. Gehan,et al.  The properties of high-dimensional data spaces: implications for exploring gene and protein expression data , 2008, Nature Reviews Cancer.

[36]  Albert Y. Zomaya,et al.  Tensor-Based Big Data Management Scheme for Dimensionality Reduction Problem in Smart Grid Systems: SDN Perspective , 2018, IEEE Transactions on Knowledge and Data Engineering.

[37]  Jian Yang,et al.  Robust Joint Feature Weights Learning Framework , 2016, IEEE Transactions on Knowledge and Data Engineering.

[38]  Nenghai Yu,et al.  Face Recognition Using Neighborhood Preserving Projections , 2005, PCM.

[39]  Terence Sim,et al.  The CMU Pose, Illumination, and Expression Database , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Witold Pedrycz,et al.  Global and local structure preserving sparse subspace learning: An iterative approach to unsupervised feature selection , 2015, Pattern Recognit..

[41]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[42]  Pichao Wang,et al.  Robust unsupervised feature selection via dual self-representation and manifold regularization , 2018, Knowl. Based Syst..

[43]  Yong Yu,et al.  Robust Subspace Segmentation by Low-Rank Representation , 2010, ICML.

[44]  Marimuthu Palaniswami,et al.  A Rapid Hybrid Clustering Algorithm for Large Volumes of High Dimensional Data , 2019, IEEE Transactions on Knowledge and Data Engineering.

[45]  Lu Wang,et al.  Orthogonal Neighborhood Preserving Embedding for Face Recognition , 2007, 2007 IEEE International Conference on Image Processing.

[46]  Shuicheng Yan,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007 .

[47]  John Wright,et al.  Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization , 2009, NIPS.

[48]  Richard H. Bartels,et al.  Algorithm 432 [C2]: Solution of the matrix equation AX + XB = C [F4] , 1972, Commun. ACM.

[49]  Habibollah Haron,et al.  Supervised, Unsupervised, and Semi-Supervised Feature Selection: A Review on Gene Selection , 2016, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[50]  Eamonn J. Keogh,et al.  Curse of Dimensionality , 2010, Encyclopedia of Machine Learning.

[51]  Andy Harter,et al.  Parameterisation of a stochastic model for human face identification , 1994, Proceedings of 1994 IEEE Workshop on Applications of Computer Vision.

[52]  Khalid Benabdeslem,et al.  Efficient Semi-Supervised Feature Selection: Constraint, Relevance, and Redundancy , 2014, IEEE Transactions on Knowledge and Data Engineering.

[53]  Yuqing He,et al.  Semi-supervised LPP algorithms for learning-to-rank-based visual search reranking , 2015, Inf. Sci..

[54]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[55]  Jia Chen,et al.  EPLS: A novel feature extraction method for migration data clustering , 2017, J. Parallel Distributed Comput..

[56]  David G. Lowe,et al.  Scalable Nearest Neighbor Algorithms for High Dimensional Data , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Hongwei Liu,et al.  Max-Margin Discriminant Projection via Data Augmentation , 2015, IEEE Transactions on Knowledge and Data Engineering.

[58]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  Adrian E. Raftery,et al.  Model-Based Clustering, Discriminant Analysis, and Density Estimation , 2002 .

[60]  Yong Yu,et al.  Robust Recovery of Subspace Structures by Low-Rank Representation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[61]  Xiaojun Chang,et al.  Adaptive Structure Discovery for Multimedia Analysis Using Multiple Features , 2019, IEEE Transactions on Cybernetics.

[62]  Lunke Fei,et al.  Low-Rank Preserving Projection Via Graph Regularized Reconstruction , 2019, IEEE Transactions on Cybernetics.

[63]  Deng Cai,et al.  Laplacian Score for Feature Selection , 2005, NIPS.

[64]  Daniel P. Robinson,et al.  Scalable Sparse Subspace Clustering by Orthogonal Matching Pursuit , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Feiping Nie,et al.  Multi-View Unsupervised Feature Selection with Adaptive Similarity and View Weight , 2017, IEEE Transactions on Knowledge and Data Engineering.

[66]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[67]  Daming Shi,et al.  TPSLVM: A Dimensionality Reduction Algorithm Based On Thin Plate Splines , 2014, IEEE Transactions on Cybernetics.

[68]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[69]  Yousef Saad,et al.  Orthogonal Neighborhood Preserving Projections: A Projection-Based Dimensionality Reduction Technique , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Zhang Yi,et al.  Constructing the L2-Graph for Robust Subspace Learning and Subspace Clustering , 2012, IEEE Transactions on Cybernetics.

[71]  Feiping Nie,et al.  Unsupervised 2D Dimensionality Reduction with Adaptive Structure Learning , 2017, Neural Computation.

[72]  Lin Wu,et al.  Beyond Low-Rank Representations: Orthogonal Clustering Basis Reconstruction with Optimized Graph Structure for Multi-view Spectral Clustering , 2017, Neural Networks.

[73]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[74]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[75]  Jinhui Tang,et al.  Unsupervised Feature Selection via Nonnegative Spectral Analysis and Redundancy Control , 2015, IEEE Transactions on Image Processing.

[76]  Rong Wang,et al.  Fast and Orthogonal Locality Preserving Projections for Dimensionality Reduction , 2017, IEEE Transactions on Image Processing.

[77]  Zi Huang,et al.  Dimensionality reduction by Mixed Kernel Canonical Correlation Analysis , 2012, Pattern Recognition.

[78]  D. Hunter,et al.  Optimization Transfer Using Surrogate Objective Functions , 2000 .

[79]  David B. Dunson,et al.  Subspace segmentation by dense block and sparse representation , 2016, Neural Networks.

[80]  Nikhil R. Pal,et al.  Unsupervised Feature Selection with Controlled Redundancy (UFeSCoR) , 2015, IEEE Transactions on Knowledge and Data Engineering.

[81]  Jiawei Han,et al.  Orthogonal Laplacianfaces for Face Recognition , 2006, IEEE Transactions on Image Processing.

[82]  Yuxiao Hu,et al.  Face recognition using Laplacianfaces , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[83]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[84]  Jiawei Han,et al.  Isometric Projection , 2007, AAAI.

[85]  Valerio Pascucci,et al.  Visualizing High-Dimensional Data: Advances in the Past Decade , 2017, IEEE Transactions on Visualization and Computer Graphics.

[86]  Junbin Gao,et al.  Laplacian Regularized Low-Rank Representation and Its Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[87]  Changsheng Xu,et al.  Inductive Robust Principal Component Analysis , 2012, IEEE Transactions on Image Processing.

[88]  Simon C. K. Shiu,et al.  Unsupervised feature selection by regularized self-representation , 2015, Pattern Recognit..

[89]  Jie Tian,et al.  Robust graph regularized unsupervised feature selection , 2018, Expert Syst. Appl..

[90]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[91]  Wai Keung Wong,et al.  Low-Rank Embedding for Robust Image Feature Extraction , 2017, IEEE Transactions on Image Processing.