Unsupervised and Semisupervised Projection With Graph Optimization

Graph-based technique is widely used in projection, clustering, and classification tasks. In this article, we propose a novel and solid framework, named unsupervised projection with graph optimization (UPGO), for both dimensionality reduction and clustering. Different from the existing algorithms which treat graph construction and projection learning as two separate steps, UPGO unifies graph construction and projection learning into a general framework. It learns the graph similarity matrix adaptively based on the relationships among the low-dimensional representations. A constraint is introduced to the Laplacian matrix to learn a structured graph which contains the clustering structure, from which the clustering results can be obtained directly without requiring any postprocessing. The structured graph achieves the ideal neighbors assignment, based on which an optimal low-dimensional subspace can be learned. Moreover, we generalize UPGO to tackle the semisupervised case, namely semisupervised projection with graph optimization (SPGO), a framework for both dimensionality reduction and classification. An efficient algorithm is derived to optimize the proposed frameworks. We provide theoretical analysis about convergence analysis, computational complexity, and parameter determination. Experimental results on real-world data sets show the effectiveness of the proposed frameworks compared with the state-of-the-art algorithms. Results also confirm the generality of the proposed frameworks.

[1]  Feiping Nie,et al.  Clustering and projected clustering with adaptive neighbors , 2014, KDD.

[2]  Ivor W. Tsang,et al.  Flexible Manifold Embedding: A Framework for Semi-Supervised and Unsupervised Dimension Reduction , 2010, IEEE Transactions on Image Processing.

[3]  Nenghai Yu,et al.  Non-negative low rank and sparse graph for semi-supervised learning , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Yong Yu,et al.  Robust Recovery of Subspace Structures by Low-Rank Representation , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Zoubin Ghahramani,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[6]  Jiawei Han,et al.  Semi-supervised Discriminant Analysis , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[7]  Feiping Nie,et al.  Semi-Supervised Multi-label Dimensionality Reduction , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[8]  K. Fan.  On a Theorem of Weyl Concerning Eigenvalues of Linear Transformations I. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Joydeep Ghosh,et al.  Cluster Ensembles --- A Knowledge Reuse Framework for Combining Multiple Partitions , 2002, J. Mach. Learn. Res..

[10]  Shuicheng Yan,et al.  Neighborhood preserving embedding , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[11]  Xuelong Li,et al.  Learning a Nonnegative Sparse Graph for Linear Regression , 2015, IEEE Transactions on Image Processing.

[12]  Shuicheng Yan,et al.  Graph embedding: a general framework for dimensionality reduction , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[13]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Ehsan Elhamifar,et al.  Sparse subspace clustering , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Shih-Fu Chang,et al.  Semi-supervised learning using greedy max-cut , 2013, J. Mach. Learn. Res..

[16]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[17]  Limei Zhang,et al.  Graph-optimized locality preserving projections , 2010, Pattern Recognit..

[18]  Jianmin Zhao,et al.  Gabor Feature Based Face Recognition Using Supervised Locality Preserving Projection , 2006, ACIVS.

[19]  Jeng-Shyang Pan,et al.  Kernel class-wise locality preserving projection , 2008, Inf. Sci..

[20]  Xuelong Li,et al.  Robust Semi-Supervised Subspace Clustering via Non-Negative Low-Rank Representation , 2016, IEEE Transactions on Cybernetics.

[21]  Yousef Saad,et al.  Orthogonal Neighborhood Preserving Projections: A Projection-Based Dimensionality Reduction Technique , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  J. Huang,et al.  Curse of dimensionality and particle filters , 2003, 2003 IEEE Aerospace Conference Proceedings (Cat. No.03TH8652).

[23]  Bo Yang,et al.  Sample-dependent graph construction with application to dimensionality reduction , 2010, Neurocomputing.

[24]  Xuan Li,et al.  Local and global structure preserving based feature selection , 2012, Neurocomputing.

[25]  Yong Yu,et al.  Robust Subspace Segmentation by Low-Rank Representation , 2010, ICML.

[26]  Rongrong Ji,et al.  Nonnegative Spectral Clustering with Discriminative Regularization , 2011, AAAI.

[27]  Pietro Perona,et al.  Self-Tuning Spectral Clustering , 2004, NIPS.

[28]  Bo Yang,et al.  A Comparative Study: Globality versus Locality for Graph Construction in Discriminant Analysis , 2014, J. Appl. Math..

[29]  Junbin Gao,et al.  Laplacian Regularized Low-Rank Representation and Its Applications , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[31]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[32]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[33]  Michael William Newman,et al.  The Laplacian spectrum of graphs , 2001 .

[34]  Xuelong Li,et al.  Flexible Affinity Matrix Learning for Unsupervised and Semisupervised Classification , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Tony Jebara,et al.  Structure preserving embedding , 2009, ICML '09.

[36]  Feiping Nie,et al.  The Constrained Laplacian Rank Algorithm for Graph-Based Clustering , 2016, AAAI.

[37]  Yan Meng,et al.  Semi-supervised manifold regularization with adaptive graph construction , 2017, Pattern Recognit. Lett..

[38]  Feiping Nie,et al.  Unsupervised feature extraction using a learned graph with clustering structure , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[39]  Feiping Nie,et al.  A New Simplex Sparse Learning Model to Measure Data Similarity for Clustering , 2015, IJCAI.

[40]  Daniel A. Spielman,et al.  Fitting a graph to vector data , 2009, ICML '09.

[41]  Xuelong Li,et al.  Unsupervised Large Graph Embedding , 2017, AAAI.

[42]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[43]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[44]  Mukund Balasubramanian,et al.  The Isomap Algorithm and Topological Stability , 2002, Science.

[45]  Feiping Nie,et al.  Unsupervised Single and Multiple Views Feature Extraction with Structured Graph , 2017, IEEE Transactions on Knowledge and Data Engineering.