Unsupervised neighborhood component analysis for clustering

In this paper, we propose a novel unsupervised distance metric learning algorithm. The proposed algorithm aims to maximize a stochastic variant of the leave-one-out K-nearest neighbor (KNN) score on unlabeled data, which performs distance metric learning and clustering simultaneously. We show that the joint distance metric learning and clustering problem is formulated as a trace optimization problem, and can be solved efficiently by an iterative algorithm. Moreover, the proposed approach can also learn a low dimensional projection of high dimensional data, thus it can serve as an unsupervised dimensionality reduction tool, which is capable of performing joint dimensionality reduction and clustering. We validate our method on a number of benchmark datasets, and the results demonstrate the effectiveness of the proposed algorithm.

[1]  Geoffrey E. Hinton,et al.  Neighbourhood Components Analysis , 2004, NIPS.

[2]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[3]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[4]  Amir Globerson,et al.  Metric Learning by Collapsing Classes , 2005, NIPS.

[5]  Charles T. Zahn,et al.  Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters , 1971, IEEE Transactions on Computers.

[6]  Misha Pavel,et al.  Adjustment Learning and Relevant Component Analysis , 2002, ECCV.

[7]  Lawrence K. Saul,et al.  Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifold , 2003, J. Mach. Learn. Res..

[8]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[9]  Jieping Ye,et al.  Discriminative K-means for Clustering , 2007, NIPS.

[10]  George W. Irwin,et al.  A novel locally regularized automatic construction method for RBF neural models , 2012, Neurocomputing.

[11]  Sheng Chen,et al.  Automatic Kernel Regression Modelling Using Combined Leave-One-Out Test Score and Regularised Orthogonal Least Squares , 2004, Int. J. Neural Syst..

[12]  I. Jolliffe.  Principal Component Analysis , 2002 .

[13]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[14]  Dale Schuurmans,et al.  Maximum Margin Clustering , 2004, NIPS.

[15]  Xin Liu,et al.  Document clustering based on non-negative matrix factorization , 2003, SIGIR.

[16]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[17]  Jieping Ye,et al.  Adaptive Distance Metric Learning for Clustering , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[19]  Chen Qin,et al.  Non-linear neighborhood component analysis based on constructive neural networks , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[20]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[21]  Jianwen Zhang,et al.  Maximin Separation Probability Clustering , 2015, AAAI.

[22]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[23]  Jorma Laaksonen,et al.  Regularized Neighborhood Component Analysis , 2007, SCIA.

[24]  Minrui Fei,et al.  A multi-output two-stage locally regularized model construction method using the extreme learning machine , 2014, Neurocomputing.

[25]  George W. Irwin,et al.  A novel automatic two-stage locally regularized classifier construction method using the extreme learning machine , 2013, Neurocomputing.