High Dimensional Statistical Inference and Random Matrices

Multivariate statistical analysis is concerned with observations on several variables which are thought to possess some degree of inter-dependence. Driven by problems in genetics and the social sciences, it first flowered in the earlier half of the last century. Subsequently, random matrix theory (RMT) developed, initially within physics, and more recently widely in mathematics. While some of the central objects of study in RMT are identical to those of multivariate statistics, statistical theory was slow to exploit the connection. However, with vast data collection ever more common, data sets now often have as many or more variables than the number of individuals observed. In such contexts, the techniques and results of RMT have much to offer multivariate statistics. The paper reviews some of the progress to date.

[1]  Karl Pearson F.R.S..  LIII. On lines and planes of closest fit to systems of points in space , 1901 .

[2]  J. Wishart.  THE GENERALISED PRODUCT MOMENT DISTRIBUTION IN SAMPLES FROM A NORMAL MULTIVARIATE POPULATION , 1928 .

[3]  H. Hotelling.  Analysis of a complex of statistical variables into principal components. , 1933 .

[4]  H. Hotelling.  Relations Between Two Sets of Variates , 1936 .

[5]  M. A. Girshick.  On the Sampling Theory of Roots of Determinantal Equations , 1939 .

[6]  R. Fisher.  THE SAMPLING DISTRIBUTION OF SOME STATISTICS OBTAINED FROM NON‐LINEAR EQUATIONS , 1939 .

[7]  P. Hsu.  ON THE DISTRIBUTION OF ROOTS OF CERTAIN DETERMINANTAL EQUATIONS , 1939 .

[8]  A. Mood.  On the Distribution of the Characteristic Roots of Normal Second-Moment Matrices , 1951 .

[9]  Harish-Chandra.  Differential Operators on a Semisimple Lie Algebra , 1957 .

[10]  E. Wigner.  On the Distribution of the Roots of Certain Symmetric Matrices , 1958 .

[11]  T. W. Anderson.  An Introduction to Multivariate Statistical Analysis , 1959 .

[12]  Freeman J. Dyson,et al.  The Threefold Way. Algebraic Structure of Symmetry Groups and Ensembles in Quantum Mechanics , 1962 .

[13]  T. W. Anderson.  ASYMPTOTIC THEORY FOR PRINCIPAL COMPONENT ANALYSIS , 1963 .

[14]  P. B. Kahn,et al.  Higher Order Spacing Distributions for a Class of Unitary Ensembles , 1964 .

[15]  A. James.  Distributions of Matrix Variates and Latent Roots Derived from Normal Samples , 1964 .

[16]  V. Marčenko,et al.  DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .

[17]  K. Wachter.  The Strong Limits of Random Matrix Spectra for Sample Matrices of Independent Elements , 1978 .

[18]  P. Menozzi,et al.  Synthetic maps of human gene frequencies in Europeans. , 1978, Science.

[19]  G. Box.  Robustness in the Strategy of Scientific Model Building. , 1979 .

[20]  K. Wachter.  The Limiting Empirical Measure of Multiple Discriminant Ratios , 1980 .

[21]  C. Itzykson.  The planar approximation , 1980 .

[22]  Richard A. Johnson,et al.  Applied Multivariate Statistical Analysis , 1983 .

[23]  R. Muirhead.  Aspects of Multivariate Statistical Theory , 1982, Wiley Series in Probability and Statistics.

[24]  J. Dauxois,et al.  Asymptotic theory for the principal component analysis of a vector random function: Some applications to statistical inference , 1982 .

[25]  Steen A. Andersson,et al.  Distribution of Eigenvalues in Multivariate Statistical Analysis , 1983 .

[26]  T. Barnett,et al.  Origins and Levels of Monthly and Seasonal Forecast Skill for United States Surface Air Temperatures Determined by Canonical Correlation Analysis , 1987 .

[27]  K. I. Gross,et al.  Total positivity, spherical series, and hypergeometric functions of matrix argu ment , 1989 .

[28]  Stephen J. Brown.  The Number of Factors in Security Returns , 1989 .

[29]  L. Cavalli-Sforza.  Genes, peoples and languages. , 1991, Scientific American.

[30]  L. R. Haff.  The Variational Form of Certain Bayes Estimators , 1991 .

[31]  Michael Biehl,et al.  Statistical mechanics of unsupervised structure recognition , 1994 .

[32]  R. Cann.  The history and geography of human genes , 1995, The Journal of Asian Studies.

[33]  C. Tracy,et al.  Level-spacing distributions and the Airy kernel , 1992, hep-th/9211141.

[34]  J. Nadal,et al.  Optimal unsupervised learning , 1994 .

[35]  J. Berger,et al.  Estimation of a Covariance Matrix Using the Reference Prior , 1994 .

[36]  D. F. Roberts,et al.  The History and Geography of Human Genes , 1996 .

[37]  C. Tracy,et al.  Mathematical Physics © Springer-Verlag 1996 On Orthogonal and Symplectic Matrix Ensembles , 1995 .

[38]  Geert Jan Bex,et al.  A Gaussian scenario for unsupervised learning , 1996 .

[39]  Craig A. Tracy,et al.  Correlation Functions, Cluster Functions, and Spacing Distributions for Random Matrices , 1998 .

[40]  K. Johansson.  Shape Fluctuations and Random Matrices , 1999, math/9903134.

[41]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[42]  P. Forrester,et al.  Classical Skew Orthogonal Polynomials and Random Matrices , 1999, solv-int/9907001.

[43]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[44]  D. Bosq.  Linear Processes in Function Spaces: Theory And Applications , 2000 .

[45]  I. Johnstone.  On the distribution of the largest eigenvalue in principal components analysis , 2001 .

[46]  R. Kass,et al.  Shrinkage Estimators for Covariance Matrices , 2001, Biometrics.

[47]  H. Knutsson,et al.  Detection of neural activity in functional MRI using canonical correlation analysis , 2001, Magnetic resonance in medicine.

[48]  A. Soshnikov.  A Note on Universality of the Distribution of the Largest Eigenvalues in Certain Sample Covariance Matrices , 2001, math/0104113.

[49]  P. Massart,et al.  Gaussian model selection , 2001 .

[50]  I. Jolliffe.  Principal Component Analysis , 2002 .

[51]  Andrew T. A. Wood,et al.  Laplace approximations for hypergeometric functions with matrix argument , 2002 .

[52]  E. Brezin,et al.  New correlation functions for random matrices and integrals over supergroups , 2002, math-ph/0208001.

[53]  P. Diaconis.  Patterns in eigenvalues: the 70th Josiah Willard Gibbs lecture , 2003 .

[54]  Noureddine El Karoui.  On the largest eigenvalue of Wishart matrices with identity covariance when n, p and p/n tend to infinity , 2003, math/0309355.

[55]  S. Péché,et al.  Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices , 2004, math/0403022.

[56]  Noureddine El Karoui.  A rate of convergence result for the largest eigenvalue of complex white Wishart matrices , 2004, math/0409610.

[57]  J. W. Silverstein,et al.  Eigenvalues of large sample covariance matrices of spiked population models , 2004, math/0408165.

[58]  Antonia Maria Tulino,et al.  Random Matrix Theory and Wireless Communications , 2004, Found. Trends Commun. Inf. Theory.

[59]  I. Johnstone,et al.  Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences , 2004, math/0410088.

[60]  Olivier Ledoit,et al.  A well-conditioned estimator for large-dimensional covariance matrices , 2004 .

[61]  M. Rattray,et al.  Principal-component-analysis eigenvalue spectra from data with symmetry-breaking structure. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[62]  Yan V. Fyodorov,et al.  On the largest singular values of random matrices with independent Cauchy entries , 2004, math/0403425.

[63]  S. Péché,et al.  Universality of local eigenvalue statistics for some sample covariance matrices , 2005 .

[64]  A. Edelman,et al.  Random matrix theory , 2005, Acta Numerica.

[65]  Jean-Philippe Bouchaud,et al.  Financial Applications of Random Matrix Theory: Old Laces and New Pieces , 2005 .

[66]  Per-Olof Persson,et al.  Numerical Methods for Eigenvalue Distributions of Random Matrices , 2005 .

[67]  Ronald W. Butler,et al.  Laplace approximations to hypergeometric functions of two matrix arguments , 2005 .

[68]  Noureddine El Karoui,et al.  Tracy-Widom limit for the largest eigenvalue of a large class of complex Wishart matrices , 2005 .

[69]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[70]  Carlos M. Carvalho,et al.  Sparse Statistical Modelling in Gene Expression Genomics , 2006 .

[71]  Runze Li,et al.  Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery , 2006, math/0602133.

[72]  Alan Edelman,et al.  The efficient evaluation of the hypergeometric function of a matrix argument , 2006, Math. Comput..

[73]  P. Deift.  Universality for mathematical and physical systems , 2006, math-ph/0603038.

[74]  Emmanuel J. Candès,et al.  Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies? , 2004, IEEE Transactions on Information Theory.

[75]  P. Hall,et al.  Properties of principal component methods for functional and longitudinal data analysis , 2006, math/0608022.

[76]  D. Paul.  ASYMPTOTICS OF SAMPLE EIGENSTRUCTURE FOR A LARGE DIMENSIONAL SPIKED COVARIANCE MODEL , 2007 .

[77]  G. Biroli,et al.  On the top eigenvalue of heavy-tailed random matrices , 2006, cond-mat/0609070.

[78]  Z. Bai,et al.  METHODOLOGIES IN SPECTRAL ANALYSIS OF LARGE DIMENSIONAL RANDOM MATRICES, A REVIEW , 2008 .

[79]  Matthew Harding,et al.  Explaining the single factor bias of arbitrage pricing models in finite samples , 2008 .