Modified Multidimensional Scaling and High Dimensional Clustering

Multidimensional scaling is an important dimension reduction tool in statistics and machine learning. Yet few theoretical results characterizing its statistical performance exist, not to mention any in high dimensions. By considering a unified framework that includes low, moderate and high dimensions, we study multidimensional scaling in the setting of clustering noisy data. Our results suggest that, the classical multidimensional scaling can be modified to further improve the quality of embedded samples, especially when the noise level increases. To this end, we propose {\it modified multidimensional scaling} which applies a nonlinear transformation to the sample eigenvalues. The nonlinear transformation depends on the dimensionality, sample size and moment of noise. We show that modified multidimensional scaling followed by various clustering algorithms can achieve exact recovery, i.e., all the cluster labels can be recovered correctly with probability tending to one. Numerical simulations and two real data applications lend strong support to our proposed methodology.

[1]  M. Fiedler.  Bounds for the determinant of the sum of hermitian matrices , 1971 .

[2]  Wang Zhou,et al.  Universality for the largest eigenvalue of sample covariance matrices with general population , 2013, 1304.5690.

[3]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[4]  Hans-Peter Kriegel,et al.  Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering , 2009, TKDD.

[5]  Damien Passemier,et al.  On determining the number of spikes in a high-dimensional spiked population model , 2011, 1104.2677.

[6]  H. Yau,et al.  On the principal components of sample covariance matrices , 2014, 1404.0788.

[7]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[8]  Emmanuel Abbe,et al.  Community detection and stochastic block models: recent developments , 2017, Found. Trends Commun. Inf. Theory.

[9]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[10]  Noureddine El Karoui.  Tracy–Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices , 2005, math/0503109.

[11]  Jianfeng Yao,et al.  Estimation of the number of spikes, possibly equal, in the high-dimensional case , 2011, J. Multivar. Anal..

[12]  Jun Yin,et al.  Anisotropic local laws for random matrices , 2014, 1410.3516.

[13]  Jun Yin,et al.  The outliers of a deformed Wigner matrix , 2012, 1207.5619.

[14]  Weichen Wang,et al.  An $\ell_{\infty}$ Eigenvector Perturbation Bound and Its Application , 2017, J. Mach. Learn. Res..

[15]  Fan Yang,et al.  A necessary and sufficient condition for edge universality at the largest singular values of covariance matrices , 2016, The Annals of Applied Probability.

[16]  Clifford Lam,et al.  Factor modeling for high-dimensional time series: inference for the number of factors , 2012, 1206.0613.

[17]  Qiang Sun,et al.  Principal Component Analysis for Big Data , 2018, Wiley StatsRef: Statistics Reference Online.

[18]  Raj Rao Nadakuditi,et al.  The singular values and vectors of low rank perturbations of large rectangular random matrices , 2011, J. Multivar. Anal..

[19]  Jun Yin,et al.  The Isotropic Semicircle Law and Deformation of Wigner Matrices , 2011, 1110.6449.

[20]  Jianqing Fan,et al.  ENTRYWISE EIGENVECTOR ANALYSIS OF RANDOM MATRICES WITH LOW EXPECTED RANK. , 2017, Annals of statistics.

[21]  G. Stewart,et al.  Matrix Perturbation Theory , 1990 .

[22]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[23]  Xiucai Ding,et al.  High dimensional deformed rectangular matrices with applications in matrix denoising , 2017, Bernoulli.

[24]  H. Yau,et al.  Isotropic local laws for sample covariance and generalized Wigner matrices , 2013, 1308.5729.

[25]  Jianqing Fan,et al.  An l∞ Eigenvector Perturbation Bound and Its Application to Robust Covariance Estimation , 2018, Journal of machine learning research : JMLR.

[26]  P. Groenen,et al.  Modern Multidimensional Scaling: Theory and Applications , 1999 .

[27]  Pasi Fränti,et al.  Fast Agglomerative Clustering Using a k-Nearest Neighbor Graph , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Nicolas Boumal,et al.  Near-Optimal Bounds for Phase Synchronization , 2017, SIAM J. Optim..

[29]  Qiang Sun,et al.  An analysis of classical multidimensional scaling with applications to clustering. , 2018, Information and inference : a journal of the IMA.

[30]  Wang Zhou,et al.  LOCAL DENSITY OF THE SPECTRUM ON THE EDGE FOR SAMPLE COVARIANCE MATRICES WITH GENERAL POPULATION , 2013 .