On Singular Values of Random Matrices

These notes are an expanded version of short courses given at the occasion of a school held in Université Paris-Est Marne-la-Vallée, 16–20 November 2009, by Djalil Chafäı, Olivier Guédon, Guillaume Lecué, Alain Pajor, and Shahar Mendelson. The central motivation is compressed sensing, involving interactions between empirical processes, high dimensional geometry, and random matrices.

[1]  Ivar Fredholm.  Sur une classe d’équations fonctionnelles , 1903 .

[2]  J. Wishart.  THE GENERALISED PRODUCT MOMENT DISTRIBUTION IN SAMPLES FROM A NORMAL MULTIVARIATE POPULATION , 1928 .

[3]  G. Carpenter.  In Providence, R. I. , 1929 .

[4]  C. Eckart,et al.  A principal axis transformation for non-hermitian matrices , 1939 .

[5]  M. A. Girshick.  On the Sampling Theory of Roots of Determinantal Equations , 1939 .

[6]  R. Fisher.  THE SAMPLING DISTRIBUTION OF SOME STATISTICS OBTAINED FROM NON‐LINEAR EQUATIONS , 1939 .

[7]  P. Hsu.  ON THE DISTRIBUTION OF ROOTS OF CERTAIN DETERMINANTAL EQUATIONS , 1939 .

[8]  A. Tikhonov.  On the stability of inverse problems , 1943 .

[9]  J. Neumann,et al.  Numerical inverting of matrices of high order , 1947 .

[10]  H. Weyl.  Inequalities between the Two Kinds of Eigenvalues of a Linear Transformation. , 1949, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[12]  A. Hoffman,et al.  The variation of the spectrum of a normal matrix , 1953 .

[13]  A. Horn.  On the eigenvalues of a matrix with prescribed singular values , 1954 .

[14]  T. W. Anderson.  An Introduction to Multivariate Statistical Analysis , 1959 .

[15]  A. James.  The Distribution of the Latent Roots of the Covariance Matrix , 1960 .

[16]  J. Lamperti.  ON CONVERGENCE OF STOCHASTIC PROCESSES , 1962 .

[17]  N. R. Goodman.  Statistical analysis based on a certain multivariate complex Gaussian distribution , 1963 .

[18]  Milton Abramowitz,et al.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , 1964 .

[19]  P. B. Kahn,et al.  Higher Order Spacing Distributions for a Class of Unitary Ensembles , 1964 .

[20]  C. Khatri.  Classical Statistical Analysis Based on a Certain Multivariate Complex Gaussian Distribution , 1965 .

[21]  J. Ginibre.  Statistical Ensembles of Complex, Quaternion, and Real Matrices , 1965 .

[22]  M. Fiedler,et al.  Matrix Inequalities , 1966 .

[23]  C. Borell.  The Brunn-Minkowski inequality in Gauss space , 1975 .

[24]  R. C. Thompson.  The behavior of eigenvalues and singular values under perturbations of restricted rank , 1976 .

[25]  T. Figiel,et al.  The dimension of almost spherical sections of convex bodies , 1976 .

[26]  Conditions for Sample-Continuity and the Central Limit Theorem , 1977 .

[27]  K. Wachter.  The Strong Limits of Random Matrix Spectra for Sample Matrices of Independent Elements , 1978 .

[28]  R. Cuninghame-Green,et al.  Applied Linear Algebra , 1979 .

[29]  Mario Bertero,et al.  The Stability of Inverse Problems , 1980 .

[30]  V. V. Buldygin,et al.  Sub-Gaussian random variables , 1980 .

[31]  S. Geman.  A Limit Theorem for the Norm of Random Matrices , 1980 .

[32]  Dag Jonsson.  Some limit theorems for the eigenvalues of a sample covariance matrix , 1982 .

[33]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[34]  E. Giné,et al.  Some Limit Theorems for Empirical Processes , 1984 .

[35]  B. Carl.  Inequalities of Bernstein-Jackson-type and the degree of compactness of operators in Banach spaces , 1985 .

[36]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[37]  Y. Gordon.  Some inequalities for Gaussian processes and applications , 1985 .

[38]  W. Johnson.  Best Constants in Moment Inequalities for Linear Combinations of Independent and Exchangeable Random Variables , 1985 .

[39]  J. W. Silverstein.  The Smallest Eigenvalue of a Large Dimensional Wishart Matrix , 1985 .

[40]  S. Smale.  On the efficiency of algorithms of analysis , 1985 .

[41]  A. Pajor,et al.  Subspaces of small codimension of finite-dimensional Banach spaces , 1986 .

[42]  Zoltán Füredi,et al.  Computing the volume is difficult , 1986, STOC '86.

[43]  V. Milman,et al.  Asymptotic Theory Of Finite Dimensional Normed Spaces , 1986 .

[44]  S. Resnick.  Extreme Values, Regular Variation, and Point Processes , 1987 .

[45]  E. Kostlan.  Complexity theory of numerical linear algebra , 1988 .

[46]  Zoltán Füredi,et al.  Approximation of the sphere by polytopes having few vertices , 1988 .

[47]  A. Edelman.  Eigenvalues and condition numbers of random matrices , 1988 .

[48]  B. Carl,et al.  Gelfand numbers of operators with values in a Hilbert space , 1988 .

[49]  Alain Pajor,et al.  Volume Ratio and Other s-Numbers of Operators Related to Local Properties of Banach Spaces , 1989 .

[50]  Jean Bourgain,et al.  On the duality problem for entropy numbers of operators , 1989 .

[51]  J. Bourgain.  Bounded orthogonal systems and the Λ(p)-set problem , 1989 .

[52]  G. Pisier.  The volume of convex bodies and Banach space geometry , 1989 .

[53]  S. Montgomery-Smith.  The distribution of Rademacher sums , 1990 .

[54]  B. Carl,et al.  Entropy, Compactness and the Approximation of Operators , 1990 .

[55]  Alain Pajor,et al.  Gelfand Numbers and Euclidean Sections of Large Dimensions , 1990 .

[56]  B. Bollobás.  THE VOLUME OF CONVEX BODIES AND BANACH SPACE GEOMETRY (Cambridge Tracts in Mathematics 94) , 1991 .

[57]  Charles R. Johnson,et al.  Topics in Matrix Analysis , 1991 .

[58]  Stanislaw J. Szarek,et al.  Condition numbers of random matrices , 1991, J. Complex..

[59]  M. Talagrand,et al.  Probability in Banach Spaces: Isoperimetry and Processes , 1991 .

[60]  G. Pisier,et al.  Non commutative Khintchine and Paley inequalities , 1991 .

[61]  Stephen J. Dilworth,et al.  The Distribution of Vector-Valued Rademacher Series , 1992 .

[62]  Y. Gordon.  Majorization of Gaussian processes and geometric applications , 1992 .

[63]  Richard P. Stanley,et al.  Some combinatorial aspects of the spectra of normally distributed random matrices , 1992 .

[64]  Bernstein inequalities for a class of random variables , 1993 .

[65]  C. Tracy,et al.  Introduction to Random Matrices , 1992, hep-th/9210073.

[66]  P. Hitczenko.  Domination inequality for martingale transforms of a Rademacher sequence , 1993 .

[67]  Z. Bai,et al.  Limit of the smallest eigenvalue of a large dimensional sample covariance matrix , 1993 .

[68]  G. Pisier.  Non-commutative vector valued Lp-spaces and completely p-summing maps , 1993, math/9306206.

[69]  M. Talagrand.  THE SUPREMUM OF SOME CANONICAL PROCESSES , 1994 .

[70]  M. Talagrand.  Sections of smooth convex bodies via majorizing measures , 1995 .

[71]  H. Massam,et al.  Craig-Sakamoto's theorem for the Wishart distributions on symmetric cones , 1995 .

[72]  E. Szemerédi,et al.  On the probability that a random ±1-matrix is singular , 1995 .

[73]  Tail and moment estimates for sums of independent random vectors with logarithmically concave tails , 1996 .

[74]  M. Talagrand.  New concentration inequalities in product spaces , 1996 .

[75]  M. Rudelson.  Random Vectors in the Isotropic Position , 1996, math/9608208.

[76]  R. Latala.  Estimation of moments of sums of independent real random variables , 1997 .

[77]  K. Ball.  An Elementary Introduction to Modern Convex Geometry , 1997 .

[78]  G. Lugosi,et al.  On Concentration-of-Measure Inequalities , 1998 .

[79]  A. Pajor.  Metric Entropy of the Grassmann Manifold , 1998 .

[80]  Mario Bertero,et al.  Introduction to Inverse Problems in Imaging , 1998 .

[81]  M. Talagrand.  Selecting a proportion of characters , 1998 .

[82]  J. W. Silverstein,et al.  No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices , 1998 .

[83]  Wolfgang Osten,et al.  Introduction to Inverse Problems in Imaging , 1999 .

[84]  Z. Bai.  METHODOLOGIES IN SPECTRAL ANALYSIS OF LARGE DIMENSIONAL RANDOM MATRICES , A REVIEW , 1999 .

[85]  K. Johansson.  Shape Fluctuations and Random Matrices , 1999, math/9903134.

[86]  F. Hiai,et al.  The semicircle law, free random variables, and entropy , 2006 .

[87]  P. Massart,et al.  About the constants in Talagrand's concentration inequalities for empirical processes , 2000 .

[88]  P. Deift.  Orthogonal Polynomials and Random Matrices: A Riemann-Hilbert Approach , 2000 .

[89]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[90]  E. Berger.  UNIFORM CENTRAL LIMIT THEOREMS (Cambridge Studies in Advanced Mathematics 63) By R. M. D UDLEY : 436pp., £55.00, ISBN 0-521-46102-2 (Cambridge University Press, 1999). , 2001 .

[91]  M. Ledoux.  The concentration of measure phenomenon , 2001 .

[92]  A. Soshnikov.  A Note on Universality of the Distribution of the Largest Eigenvalues in Certain Sample Covariance Matrices , 2001, math/0104113.

[93]  Jianhong Shen.  On the singular values of Gaussian random matrices , 2001 .

[94]  C. Tracy,et al.  Distribution Functions for Largest Eigenvalues and Their Applications , 2002, math-ph/0210034.

[95]  P. MassartLedoux,et al.  Concentration Inequalities Using the Entropy Method , 2002 .

[96]  E. Rio.  Une inégalité de Bennett pour les maxima de processus empiriques , 2002 .

[97]  O. Bousquet.  A Bennett concentration inequality and its application to suprema of empirical processes , 2002 .

[98]  A. Edelman,et al.  Matrix models for beta ensembles , 2002, math-ph/0206043.

[99]  Gábor Lugosi,et al.  Concentration Inequalities , 2008, COLT.

[100]  Shang-Hua Teng,et al.  Smoothed Analysis (Motivation and Discrete Models) , 2003, WADS.

[101]  Increasing subsequences and the hard-to-soft edge transition in matrix ensembles , 2002, math-ph/0205007.

[102]  O. Bousquet.  Concentration Inequalities for Sub-Additive Functions Using the Entropy Method , 2003 .

[103]  S. Thorbjørnsen,et al.  Random matrices with complex Gaussian entries , 2003 .

[104]  S. Péché,et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices , 2004, math/0403022.

[105]  Jean-Marc Azaïs,et al.  Upper and Lower Bounds for the Tails of the Distribution of the Condition Number of a Gaussian Matrix , 2005, SIAM J. Matrix Anal. Appl..

[106]  Tharmalingam Ratnarajah,et al.  Eigenvalues and Condition Numbers of Complex Random Matrices , 2005, SIAM J. Matrix Anal. Appl..

[107]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[108]  Michel Ledoux,et al.  Differential Operators and Spectral Distributions of Invariant Ensembles from the Classical Orthogonal Polynomials. The Continuous Case , 2004 .

[109]  S. Artstein,et al.  Duality of metric entropy , 2004 .

[110]  R. Lata,et al.  SOME ESTIMATES OF NORMS OF RANDOM MATRICES , 2004 .

[111]  P. Loubaton,et al.  The empirical distribution of the eigenvalues of a Gram matrix with a given variance profile , 2004, math/0411333.

[112]  S. Mendelson.  On weakly bounded empirical processes , 2005, math/0512554.

[113]  T. Tao,et al.  Inverse Littlewood-Offord theorems and the condition number of random discrete matrices , 2005, math/0511215.

[114]  L. Trefethen,et al.  Spectra and pseudospectra : the behavior of nonnormal matrices and operators , 2005 .

[115]  J. W. Silverstein,et al.  Analysis of the limiting spectral distribution of large dimensional information-plus-noise type matrices , 2005 .

[116]  Alan Edelman,et al.  Tails of Condition Number Distributions , 2005, SIAM J. Matrix Anal. Appl..

[117]  Zizhong Chen,et al.  Condition Numbers of Gaussian Random Matrices , 2005, SIAM J. Matrix Anal. Appl..

[118]  Alan Edelman,et al.  Eigenvalues of Hermite and Laguerre ensembles: large beta asymptotics , 2005 .

[119]  Daniel A. Spielman.  The Smoothed Analysis of Algorithms , 2005, FCT.

[120]  L. Trefethen.  Spectra and pseudospectra , 2005 .

[121]  Yan V. Fyodorov,et al.  On the largest singular values of random matrices with independent Cauchy entries , 2004, math/0403425.

[122]  S. Péché,et al.  Universality of local eigenvalue statistics for some sample covariance matrices , 2005 .

[123]  G. Golub,et al.  Inverse Eigenvalue Problems: Theory, Algorithms, and Applications , 2005 .

[124]  David L. Donoho,et al.  Neighborly Polytopes And Sparse Solution Of Underdetermined Linear Equations , 2005 .

[125]  V. Bentkus.  A Lyapunov-type Bound in Rd , 2005 .

[126]  E. Candès,et al.  Stable signal recovery from incomplete and inaccurate measurements , 2005, math/0503066.

[127]  M. Rudelson,et al.  Lp-moments of random vectors via majorizing measures , 2005, math/0507023.

[128]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[129]  E. Rio,et al.  Concentration around the mean for maxima of empirical processes , 2005, math/0506594.

[130]  T. Tao,et al.  On the singularity probability of random Bernoulli matrices , 2005, math/0501313.

[131]  M. Rudelson.  Invertibility of random matrices: norm of the inverse , 2005, math/0507024.

[132]  M. Stephanov,et al.  Random Matrices , 2005, hep-ph/0509286.

[133]  M. Rudelson,et al.  Smallest singular value of random matrices and geometry of random polytopes , 2005 .

[134]  P. Bartlett,et al.  Empirical minimization , 2006 .

[135]  S. Mendelson,et al.  On singular values of matrices with independent rows , 2006 .

[136]  Nathan Linial,et al.  How Neighborly Can a Centrally Symmetric Polytope Be? , 2006, Discret. Comput. Geom..

[137]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[138]  G. Paouris.  Concentration of mass on convex bodies , 2006 .

[139]  Mark Rudelson,et al.  Lower estimates for the singular values of random matrices , 2006 .

[140]  Alexander Soshnikov,et al.  Poisson Statistics for the Largest Eigenvalues in Random Matrix Ensembles , 2006 .

[141]  S. Mendelson,et al.  Uniform Uncertainty Principle for Bernoulli and Subgaussian Ensembles , 2006, math/0608665.

[142]  P. Deift.  Universality for mathematical and physical systems , 2006, math-ph/0603038.

[143]  V. Koltchinskii.  Local Rademacher complexities and oracle inequalities in risk minimization , 2006, 0708.0083.

[144]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[145]  Franck Barthe,et al.  The BrunnMinkowski theorem and related geometric and functional inequalities , 2006 .

[146]  T. Tao,et al.  On random ±1 matrices: Singularity and determinant , 2006 .

[147]  T. Tao,et al.  RANDOM MATRICES: THE CIRCULAR LAW , 2007, 0708.2895.

[148]  R. Adamczak.  A tail inequality for suprema of unbounded empirical processes with applications to Markov chains , 2007, 0709.3110.

[149]  M. Rudelson,et al.  The Littlewood-Offord problem and invertibility of random matrices , 2007, math/0703503.

[150]  B. Klartag,et al.  A Berry-Esseen type inequality for convex bodies with an unconditional basis , 2007, 0705.0832.

[151]  Shahar Mendelson,et al.  Gaussian averages of interpolated bodies and applications to approximate reconstruction , 2007, J. Approx. Theory.

[152]  S. Mendelson,et al.  Subspaces and Orthogonal Decompositions Generated by Bounded Orthogonal Systems , 2007 .

[153]  Noureddine El Karoui.  Tracy–Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices , 2005, math/0503109.

[154]  J. W. Silverstein,et al.  On the empirical distribution of eigenvalues of large dimensional information-plus-noise-type matrices , 2007 .

[155]  S. Péché.  Universality results for largest eigenvalues of some sample covariance matrix ensembles , 2007, 0705.1701.

[156]  B. Klartag.  A central limit theorem for convex sets , 2006, math/0605014.

[157]  S. Mendelson,et al.  Reconstruction and Subgaussian Operators in Asymptotic Geometric Analysis , 2007 .

[158]  S. Mendelson,et al.  Majorizing measures and proportional subsets of bounded orthonormal systems , 2008, 0801.3556.

[159]  M. Rudelson,et al.  On sparse reconstruction from Fourier and Gaussian measurements , 2008 .

[160]  R. DeVore,et al.  Compressed sensing and best k-term approximation , 2008 .

[161]  M. Rudelson,et al.  The least singular value of a random square matrix is O(n−1/2) , 2008, 0805.3407.

[162]  Francois Germinet,et al.  On the singularity of random matrices with independent entries , 2008 .

[163]  Ohad N. Feldheim,et al.  A Universality Result for the Smallest Eigenvalues of Certain Sample Covariance Matrices , 2008, 0812.1961.

[164]  Alexander E. Litvak,et al.  Smallest singular value of random matrices with independent columns , 2008 .

[165]  Ioana Dumitriu,et al.  Distributions of the Extreme Eigenvaluesof Beta-Jacobi Random Matrices , 2008, SIAM J. Matrix Anal. Appl..

[166]  M. Rudelson,et al.  The smallest singular value of a random rectangular matrix , 2008, 0802.3956.

[167]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[168]  Jelena Kovacevic,et al.  An Introduction to Frames , 2008, Found. Trends Signal Process..

[169]  E. Candès.  The restricted isometry property and its implications for compressed sensing , 2008 .

[170]  Terence Tao,et al.  Random matrices: Universality of ESDs and the circular law , 2008, 0807.4898.

[171]  R. Adamczak,et al.  Restricted Isometry Property of Matrices with Independent Columns and Neighborly Polytopes by Random Sampling , 2009, 0904.4723.

[172]  T. Tao,et al.  Random Matrices: the Distribution of the Smallest Singular Values , 2009, 0903.0614.

[173]  J. W. Silverstein,et al.  Spectral Analysis of Large Dimensional Random Matrices , 2009 .

[174]  Amir Dembo,et al.  Spectral Measure of Heavy Tailed Band and Covariance Random Matrices , 2008, 0811.1587.

[175]  J. Azaïs,et al.  Level Sets and Extrema of Random Processes and Fields , 2009 .

[176]  S. Foucart,et al.  Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 , 2009 .

[177]  R. Adamczak,et al.  Quantitative estimates of the convergence of the empirical covariance matrix in log-concave ensembles , 2009, 0903.2323.

[178]  H. Rauhut.  Compressive Sensing and Structured Random Matrices , 2009 .

[179]  S. P'ech'e,et al.  The largest eigenvalues of sample covariance matrices for a spiked population: Diagonal case , 2008, 0812.2320.

[180]  Jean Bourgain,et al.  On the singularity probability of discrete random matrices , 2009, 0905.0461.

[181]  Holger Rauhut,et al.  The Gelfand widths of ℓp-balls for 0 , 2010, ArXiv.

[182]  Ming-Jun Lai,et al.  Sparse recovery with pre-Gaussian random matrices , 2010 .

[183]  P. Forrester.  Log-Gases and Random Matrices , 2010 .

[184]  Terence Tao,et al.  Smooth analysis of the condition number and the least singular value , 2008, Math. Comput..

[185]  Robert H. Halstead,et al.  Matrix Computations , 2011, Encyclopedia of Parallel Computing.

[186]  J. Ramírez,et al.  Beta ensembles, stochastic Airy spectrum, and a diffusion , 2006, math/0607331.

[187]  Alex Gittens,et al.  TAIL BOUNDS FOR ALL EIGENVALUES OF A SUM OF RANDOM MATRICES , 2011, 1104.4513.

[188]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[189]  J. Demmel.  The Probability That a Numerical, Analysis Problem Is Difficult , 2013 .

[190]  H. König,et al.  Asymptotic Geometric Analysis , 2015 .

[191]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[192]  Teodoro Collin.  RANDOM MATRIX THEORY , 2016 .