LDA-Based Clustering Algorithm and Its Application to an Unsupervised Feature Extraction

Research has shown fuzzy c-means (FCM) clustering to be a powerful tool to partition samples into different categories. However, the objective function of FCM is based only on the sum of distances of samples to their cluster centers, which is equal to the trace of the within-cluster scatter matrix. In this study, we propose a clustering algorithm based on both within- and between-cluster scatter matrices, extended from linear discriminant analysis (LDA), and its application to an unsupervised feature extraction (FE). Our proposed methods comprise between- and within-cluster scatter matrices modified from the between- and within-class scatter matrices of LDA. The scatter matrices of LDA are special cases of our proposed unsupervised scatter matrices. The results of experiments on both synthetic and real data show that the proposed clustering algorithm can generate similar or better clustering results than 11 popular clustering algorithms: K-means, K-medoid, FCM, the Gustafson-Kessel, Gath-Geva, possibilistic c-means (PCM), fuzzy PCM, possibilistic FCM, fuzzy compactness and separation, a fuzzy clustering algorithm based on a fuzzy treatment of finite mixtures of multivariate Student's t distributions algorithms, and a fuzzy mixture of the Student's t factor analyzers model. The results also show that the proposed FE outperforms principal component analysis and independent component analysis.

[1]  Aljoscha C. Neubauer,et al.  The Mental Speed-IQ Relationship: Unitary or Modular?. , 1996 .

[2]  Giovanni Soda,et al.  Embedded Map Projection for Dimensionality Reduction-Based Similarity Search , 2008, SSPR/SPR.

[3]  Gabriele Moser,et al.  Extraction of Spectral Channels From Hyperspectral Images for Classification Purposes , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Larry Nazareth,et al.  A family of variable metric updates , 1977, Math. Program..

[5]  Sotirios Chatzis,et al.  Robust fuzzy clustering using mixtures of Student's-t distributions , 2008, Pattern Recognit. Lett..

[6]  C. S. George Lee,et al.  Neural fuzzy systems: a neuro-fuzzy synergism to intelligent systems , 1996 .

[7]  Uzay Kaymak,et al.  Improved covariance estimation for Gustafson-Kessel clustering , 2002, 2002 IEEE World Congress on Computational Intelligence. 2002 IEEE International Conference on Fuzzy Systems. FUZZ-IEEE'02. Proceedings (Cat. No.02CH37291).

[8]  Sotirios Chatzis,et al.  Factor Analysis Latent Subspace Modeling and Robust Fuzzy Clustering Using $t$-Distributions , 2009, IEEE Transactions on Fuzzy Systems.

[9]  Stephen P. Boyd,et al.  Optimal kernel selection in Kernel Fisher discriminant analysis , 2006, ICML.

[10]  Stan Matwin,et al.  Filtering Multi-Instance Problems to Reduce Dimensionality in Relational Learning , 2004, Journal of Intelligent Information Systems.

[11]  尹中航,et al.  Fuzzy Clustering with Novel Separable Criterion , 2006 .

[12]  James C. Bezdek,et al.  Optimal Fuzzy Partitions: A Heuristic for Estimating the Parameters in a Mixture of Normal Distributions , 1975, IEEE Transactions on Computers.

[13]  Denis Hamad,et al.  K-means Clustering Algorithm in Projected Spaces , 2006, 2006 9th International Conference on Information Fusion.

[14]  Bor-Chen Kuo,et al.  Regularized feature extractions for hyperspectral data classification , 2003, IGARSS 2003. 2003 IEEE International Geoscience and Remote Sensing Symposium. Proceedings (IEEE Cat. No.03CH37477).

[15]  Fionn Murtagh,et al.  Overcoming the Curse of Dimensionality in Clustering by Means of the Wavelet Transform , 2000, Comput. J..

[16]  Keinosuke Fukunaga,et al.  Introduction to Statistical Pattern Recognition , 1972 .

[17]  I. Jolliffe.  Principal Component Analysis , 2002 .

[18]  Geoffrey E. Hinton,et al.  The EM algorithm for mixtures of factor analyzers , 1996 .

[19]  Rüdiger Dillmann,et al.  Fast and Robust Feature-based Recognition of Multiple Objects , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[20]  Donald Gustafson,et al.  Fuzzy clustering with a fuzzy covariance matrix , 1978, 1978 IEEE Conference on Decision and Control including the 17th Symposium on Adaptive Processes.

[21]  James M. Keller,et al.  A possibilistic fuzzy c-means clustering algorithm , 2005, IEEE Transactions on Fuzzy Systems.

[22]  Hidetomo Ichihashi,et al.  Regularized linear fuzzy clustering and probabilistic PCA mixture models , 2005, IEEE Transactions on Fuzzy Systems.

[23]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2004 .

[24]  Chia-Wei Hsu,et al.  A Linear Feature Extraction for Multiclass Classification Problems Based on Class Mean and Covariance Discriminant Information , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .

[26]  James C. Bezdek,et al.  Pattern Recognition with Fuzzy Objective Function Algorithms , 1981, Advanced Applications in Pattern Recognition.

[27]  Paulo J. G. Lisboa,et al.  Fuzzy systems in medicine , 2000 .

[28]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[29]  James C. Bezdek,et al.  A mixed c-means clustering model , 1997, Proceedings of 6th International Fuzzy Systems Conference.

[30]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[31]  Geoffrey J. McLachlan,et al.  Extension of the mixture of factor analyzers model to incorporate the multivariate t-distribution , 2007, Comput. Stat. Data Anal..

[32]  M. Omair Ahmad,et al.  Optimizing the kernel in the empirical feature space , 2005, IEEE Transactions on Neural Networks.

[33]  Jian Yu,et al.  A novel fuzzy clustering algorithm based on a fuzzy scatter matrix with optimality tests , 2005, Pattern Recognit. Lett..

[34]  Ron Kohavi,et al.  Feature Subset Selection Using the Wrapper Method: Overfitting and Dynamic Search Space Topology , 1995, KDD.

[35]  Jorge Nocedal,et al.  An interior algorithm for nonlinear optimization that combines line search and trust region steps , 2006, Math. Program..

[36]  N. Cressie,et al.  A dimension-reduced approach to space-time Kalman filtering , 1999 .

[37]  Bor-Chen Kuo,et al.  Nonparametric weighted feature extraction for classification , 2004, IEEE Transactions on Geoscience and Remote Sensing.

[38]  Dat Tran,et al.  A robust clustering approach to fuzzy Gaussian mixture models for speaker identification , 1999, 1999 Third International Conference on Knowledge-Based Intelligent Information Engineering Systems. Proceedings (Cat. No.99TH8410).

[39]  Peter J. Rousseeuw,et al.  Fuzzy clustering using scatter matrices , 1996 .

[40]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[41]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[42]  Ludmila I. Kuncheva,et al.  Evaluation of Stability of k-Means Cluster Ensembles with Respect to Random Initialization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Giovanni Soda,et al.  Nonlinear Embedded Map Projection for Dimensionality Reduction , 2009, ICIAP.

[44]  Gerardo Beni,et al.  A Validity Measure for Fuzzy Clustering , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  E. Oja,et al.  Independent Component Analysis , 2013 .

[46]  Balazs Feil,et al.  Fuzzy Clustering and Data Analysis Toolbox For Use with Matlab , 2005 .

[47]  S. T. Buckland,et al.  An Introduction to the Bootstrap. , 1994 .

[48]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[49]  Hidetomo Ichihashi,et al.  Fuzzy PCA-Guided Robust $k$-Means Clustering , 2010, IEEE Transactions on Fuzzy Systems.

[50]  Yoram J. Kaufman,et al.  Principal Component Analysis of Remote Sensing of Aerosols Over Oceans , 2007, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[52]  Constantin V. Negoita,et al.  On Fuzzy Systems , 1978 .

[53]  Bor-Chen Kuo,et al.  Regularized Feature Extractions and Support Vector Machines for Hyperspectral Image Data Classification , 2005, KES.

[54]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[55]  James M. Keller,et al.  A possibilistic approach to clustering , 1993, IEEE Trans. Fuzzy Syst..

[56]  Huan Liu,et al.  Hybrid Search of Feature Subsets , 1998, PRICAI.