Regularized soft K-means for discriminant analysis

Traditionally unsupervised dimensionality reduction methods may not necessarily improve the separability of the data resided in different clusters due to ignorance of the inherent relationship between subspace selection and clustering. It is known that soft clustering using fuzzy c-means or its variants can provide a better and more meaningful data partition than hard clustering, which motivates us to develop a novel entropy regularized soft K-means algorithm for discriminant analysis (ResKmeans) in this paper. ResKmeans performs soft clustering and subspace selection simultaneously and thus gives rise to a generalized linear discriminant analysis (GELDA) which captures both the intra-cluster compactness and the inter-cluster separability. Furthermore, we clarify both the relationship between GELDA and conventional LDA and the inherent relationship between subspace selection and soft clustering. Experimental results on real-world data sets show ResKmeans is superior to other popular clustering algorithms.

[1]  Jieping Ye,et al.  Discriminative K-means for Clustering , 2007, NIPS.

[2]  Daoqiang Zhang,et al.  Fast and robust fuzzy c-means clustering algorithms incorporating local information for image segmentation , 2007, Pattern Recognit..

[3]  W. Peizhuang.  Pattern Recognition with Fuzzy Objective Function Algorithms (James C. Bezdek) , 1983 .

[4]  Rong Jin,et al.  A Boosting Framework for Visuality-Preserving Distance Metric Learning and Its Application to Medical Image Retrieval , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Bo Yang,et al.  Sample-dependent graph construction with application to dimensionality reduction , 2010, Neurocomputing.

[6]  Jiawei Han,et al.  Locally Consistent Concept Factorization for Document Clustering , 2011, IEEE Transactions on Knowledge and Data Engineering.

[7]  I. Jolliffe.  Principal Component Analysis , 2002 .

[8]  Juan Miguel Ortiz-de-Lazcano-Lobato,et al.  Principal Components Analysis Competitive Learning , 2003, Neural Computation.

[9]  Stephen Lin,et al.  Graph Embedding and Extensions: A General Framework for Dimensionality Reduction , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Song-can Chen,et al.  Kernel-based fuzzy and possibilistic c-means clustering , 2003 .

[11]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[12]  Sadaaki Miyamoto,et al.  Fuzzy c-means as a regularization and maximum entropy approach , 1997 .

[13]  Chun Chen,et al.  Constrained Laplacian Eigenmap for dimensionality reduction , 2010, Neurocomputing.

[14]  J. Friedman.  Regularized Discriminant Analysis , 1989 .

[15]  Rajesh N. Davé,et al.  Robust fuzzy clustering of relational data , 2002, IEEE Trans. Fuzzy Syst..

[16]  Takeo Kanade,et al.  Discriminative cluster analysis , 2006, ICML.

[17]  Xiaofei He,et al.  Locality Preserving Projections , 2003, NIPS.

[18]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[19]  Hui Xiong,et al.  Enhancing semi-supervised clustering: a feature projection perspective , 2007, KDD '07.

[20]  David J. Kriegman,et al.  Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection , 1996, ECCV.

[21]  Lawrence K. Saul,et al.  Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifold , 2003, J. Mach. Learn. Res..

[22]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[23]  Jieping Ye,et al.  Nonlinear adaptive distance metric learning for clustering , 2007, KDD '07.

[24]  Jieping Ye,et al.  Canonical Correlation Analysis for Multilabel Classification: A Least-Squares Formulation, Extensions, and Analysis , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Lawrence O. Hall,et al.  Fast Accurate Fuzzy Clustering through Data Reduction , 2003 .

[26]  Isak Gath,et al.  Unsupervised Optimal Fuzzy Clustering , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Daoqiang Zhang,et al.  Semi-supervised clustering with metric learning: An adaptive kernel method , 2010, Pattern Recognit..

[28]  Jieping Ye,et al.  Adaptive Distance Metric Learning for Clustering , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Kohei Inoue,et al.  Robust Kernel Fuzzy Clustering , 2005, FSKD.