Robust unsupervised feature selection via matrix factorization

We proposed a robust unsupervised method to remove redundant and irrelevant features.Both the cluster centers and the sparse representation are predicted.The feature selection and clustering are performed simultaneously.An efficient iterative update algorithm based on ADMM is proposed.Superiority over seven existing methods is established for five data sets. Dimensionality reduction is a challenging task for high-dimensional data processing in machine learning and data mining. It can help to reduce computation time, save storage space and improve the performance of learning algorithms. As an effective dimension reduction technique, unsupervised feature selection aims at finding a subset of features to retain the most relevant information. In this paper, we propose a novel unsupervised feature selection method, called Robust Unsupervised Feature Selection via Matrix Factorization (RUFSM), in which robust discriminative feature selection and robust clustering are performed simultaneously under l2, 1-norm while the local manifold structures of data are preserved. The advantages of this work are three-fold. Firstly, both the latent orthogonal cluster centers and the sparse representation of the projected data points based on matrix factorization are predicted for selecting robust discriminative features. Secondly, the feature selection and the clustering are performed simultaneously to guarantee an overall optimum. Thirdly, an efficient iterative update algorithm, which is based on Alternating Direction Method of Multipliers (ADMM), is used for RUFSM optimization. Compared with several state-of-the-art unsupervised feature selection methods, the proposed algorithm comes with better clustering performance for almost all datasets we have experimented with here.

[1]  Yong Yu,et al.  Robust Subspace Segmentation by Low-Rank Representation , 2010, ICML.

[2]  Deng Cai,et al.  Unsupervised feature selection for multi-cluster data , 2010, KDD.

[3]  Huan Liu,et al.  Spectral feature selection for supervised and unsupervised learning , 2007, ICML '07.

[4]  ChengXiang Zhai,et al.  Robust Unsupervised Feature Selection , 2013, IJCAI.

[5]  Yong Peng,et al.  Enhanced low-rank representation via sparse manifold adaption for semi-supervised learning , 2015, Neural Networks.

[6]  René Vidal,et al.  Sparse Subspace Clustering: Algorithm, Theory, and Applications , 2012, IEEE transactions on pattern analysis and machine intelligence.

[7]  José M. Bioucas-Dias,et al.  An Augmented Lagrangian Approach to the Constrained Optimization Formulation of Imaging Inverse Problems , 2009, IEEE Transactions on Image Processing.

[8]  Jianbo Shi,et al.  Multiclass spectral clustering , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[9]  Jennifer G. Dy,et al.  From Transformation-Based Dimensionality Reduction to Feature Selection , 2010, ICML.

[10]  Chun Chen,et al.  Graph Regularized Sparse Coding for Image Representation , 2011, IEEE Transactions on Image Processing.

[11]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Jitendra Malik,et al.  Normalized Cuts and Image Segmentation , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Witold Pedrycz,et al.  Subspace learning for unsupervised feature selection via matrix factorization , 2015, Pattern Recognit..

[14]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[15]  David J. Kriegman,et al.  Video-based face recognition using probabilistic appearance manifolds , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[16]  Le Song,et al.  Supervised feature selection via dependence estimation , 2007, ICML '07.

[17]  Masoud Nikravesh,et al.  Feature Extraction: Foundations and Applications (Studies in Fuzziness and Soft Computing) , 2006 .

[18]  Feiping Nie,et al.  Robust Manifold Nonnegative Matrix Factorization , 2014, ACM Trans. Knowl. Discov. Data.

[19]  Xuelong Li,et al.  Semisupervised Dimensionality Reduction and Classification Through Virtual Label Regression , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[20]  Feiping Nie,et al.  The Constrained Laplacian Rank Algorithm for Graph-Based Clustering , 2016, AAAI.

[21]  Jing Liu,et al.  Unsupervised Feature Selection Using Nonnegative Spectral Analysis , 2012, AAAI.

[22]  Deng Cai,et al.  Laplacian Score for Feature Selection , 2005, NIPS.

[23]  Jiawei Han,et al.  Locally Consistent Concept Factorization for Document Clustering , 2011, IEEE Transactions on Knowledge and Data Engineering.

[24]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Jianfeng Gao,et al.  Scalable training of L1-regularized log-linear models , 2007, ICML '07.

[26]  Yi Yang,et al.  A Convex Formulation for Semi-Supervised Multi-Label Feature Selection , 2014, AAAI.

[27]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[28]  Nathan Halko,et al.  Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions , 2009, SIAM Rev..

[29]  Feiping Nie,et al.  Trace Ratio Criterion for Feature Selection , 2008, AAAI.

[30]  Rong Wang,et al.  Robust 2DPCA With Non-greedy $\ell _{1}$ -Norm Maximization for Image Analysis , 2015, IEEE Transactions on Cybernetics.

[31]  Shuicheng Yan,et al.  Similarity preserving low-rank representation for enhanced data representation and effective subspace learning , 2014, Neural Networks.

[32]  Simon C. K. Shiu,et al.  Unsupervised feature selection by regularized self-representation , 2015, Pattern Recognit..

[33]  Yueting Zhuang,et al.  Graph Regularized Feature Selection with Data Reconstruction , 2016, IEEE Transactions on Knowledge and Data Engineering.

[34]  T. Wieczorek,et al.  Comparison of feature ranking methods based on information entropy , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[35]  Yang Liu,et al.  Manifold regularized multi-view feature selection for social image annotation , 2016, Neurocomputing.

[36]  Feiping Nie,et al.  Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization , 2010, NIPS.

[37]  Yihong Gong,et al.  Document clustering by concept factorization , 2004, SIGIR '04.

[38]  Carla E. Brodley,et al.  Feature Selection for Unsupervised Learning , 2004, J. Mach. Learn. Res..

[39]  Mark W. Schmidt,et al.  Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches , 2007, ECML.

[40]  Thomas S. Huang,et al.  Graph Regularized Nonnegative Matrix Factorization for Data Representation. , 2011, IEEE transactions on pattern analysis and machine intelligence.

[41]  Volker Roth,et al.  The generalized LASSO , 2004, IEEE Transactions on Neural Networks.

[42]  Junmo Kim,et al.  Unsupervised Simultaneous Orthogonal basis Clustering Feature Selection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Masoud Nikravesh,et al.  Feature Extraction - Foundations and Applications , 2006, Feature Extraction.

[44]  Jiawei Han,et al.  Generalized Fisher Score for Feature Selection , 2011, UAI.

[45]  Lior Wolf,et al.  Feature selection for unsupervised and supervised inference: the emergence of sparsity in a weighted-based approach , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[46]  Zi Huang,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence ℓ2,1-Norm Regularized Discriminative Feature Selection for Unsupervised Learning , 2022 .