Localization on low-order eigenvectors of data matrices

Eigenvector localization refers to the situation when most of the components of an eigenvector are zero or near-zero. This phenomenon has been observed on eigenvectors associated with extremal eigenvalues, and in many of those cases it can be meaningfully interpreted in terms of "structural heterogeneities" in the data. For example, the largest eigenvectors of adjacency matrices of large complex networks often have most of their mass localized on high-degree nodes; and the smallest eigenvectors of the Laplacians of such networks are often localized on small but meaningful community-like sets of nodes. Here, we describe localization associated with low-order eigenvectors, i.e., eigenvectors corresponding to eigenvalues that are not extremal but that are "buried" further down in the spectrum. Although we have observed it in several unrelated applications, this phenomenon of low-order eigenvector localization defies common intuitions and simple explanations, and it creates serious difficulties for the applicability of popular eigenvector-based machine learning and data analysis tools. After describing two examples where low-order eigenvector localization arises, we present a very simple model that qualitatively reproduces several of the empirically-observed results. This model suggests certain coarse structural similarities among the seemingly-unrelated applications where we have observed low-order eigenvector localization, and it may be used as a diagnostic tool to help extract insight from data graphs when such low-order eigenvector localization is present.

[1]  Ulrike von Luxburg,et al.  A tutorial on spectral clustering , 2007, Stat. Comput..

[2]  K. T. Poole,et al.  Congress: A Political-Economic History of Roll Call Voting , 1997 .

[3]  Paul Van Dooren,et al.  Extracting spatial information from networks with low-order eigenvectors , 2011, ArXiv.

[4]  M. Newman,et al.  Finding community structure in networks using the eigenvectors of matrices. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  Alex Pothen,et al.  PARTITIONING SPARSE MATRICES WITH EIGENVECTORS OF GRAPHS* , 1990 .

[6]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[7]  Shang-Hua Teng,et al.  Spectral partitioning works: planar graphs and finite element meshes , 1996, Proceedings of 37th Conference on Foundations of Computer Science.

[8]  Nisheeth K. Vishnoi,et al.  A Spectral Algorithm for Improving Graph Partitions with Applications to Exploring Data Graphs Locally , 2009 .

[9]  J. A. Díaz-García,et al.  SENSITIVITY ANALYSIS IN LINEAR REGRESSION , 2022 .

[10]  Paul B. Slater,et al.  Hubs and Clusters in the Evolving U. S. Internal Migration Network , 2008, ArXiv.

[11]  Christos Faloutsos,et al.  Graph mining: Laws, generators, and algorithms , 2006, CSUR.

[12]  Jure Leskovec,et al.  Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters , 2008, Internet Math..

[13]  S. N. Dorogovtsev,et al.  Spectra of complex networks. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[14]  Ann B. Lee,et al.  Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Robert S. Strichartz,et al.  Localized Eigenfunctions: Here You See Them, There You Don't , 2009, 0909.0783.

[16]  Nicolas Le Roux,et al.  Learning Eigenfunctions Links Spectral Embedding and Kernel PCA , 2004, Neural Computation.

[17]  John B. Moore,et al.  Singular Value Decomposition , 1994 .

[18]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[19]  Luca Trevisan,et al.  Max cut and the smallest eigenvalue , 2008, STOC '09.

[20]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[21]  S. Chatterjee.  Sensitivity analysis in linear regression , 1988 .

[22]  K. Goh,et al.  Spectra and eigenvectors of scale-free networks. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[23]  João Pedro Hespanha,et al.  Effective resistance of Gromov-hyperbolic graphs: Application to asymptotic sensor network problems , 2007, 2007 46th IEEE Conference on Decision and Control.

[24]  V. N. Bogaevski,et al.  Matrix Perturbation Theory , 1991 .

[25]  Michael I. Jordan,et al.  On Spectral Clustering: Analysis and an algorithm , 2001, NIPS.

[26]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[28]  Jianbo Shi,et al.  A Random Walks View of Spectral Segmentation , 2001, AISTATS.

[29]  Fan Chung Graham,et al.  Local Graph Partitioning using PageRank Vectors , 2006, 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06).

[30]  Neil Muller,et al.  Singular Value Decomposition, Eigenfaces, and 3D Reconstructions , 2004, SIAM Rev..

[31]  Jukka-Pekka Onnela,et al.  Community Structure in Time-Dependent, Multiscale, and Multiplex Networks , 2009, Science.

[32]  Petros Drineas,et al.  CUR matrix decompositions for improved data analysis , 2009, Proceedings of the National Academy of Sciences.

[33]  Alan M. Frieze,et al.  Random graphs , 2006, SODA '06.

[34]  P. Mucha,et al.  Party Polarization in Congress: A Social Networks Approach , 2009, 0907.3509.

[35]  Nisheeth K. Vishnoi,et al.  A local spectral method for graphs: with applications to improving graph partitions and exploring data graphs locally , 2009, J. Mach. Learn. Res..

[36]  Shang-Hua Teng,et al.  Nearly-linear time algorithms for graph partitioning, graph sparsification, and solving linear systems , 2003, STOC '04.

[37]  P. Barooah,et al.  Graph Effective Resistance and Distributed Control: Spectral Properties and Applications , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[38]  Yair Weiss,et al.  Segmentation using eigenvectors: a unifying view , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[39]  A. Barabasi,et al.  Spectra of "real-world" graphs: beyond the semicircle law. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[40]  M. Mitrovic,et al.  Spectral and dynamical properties in classes of sparse networks with mesoscopic inhomogeneities. , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.