Halting Time is Predictable for Large Models: A Universality Property and Average-Case Analysis
暂无分享,去创建一个
Fabian Pedregosa | Bart van Merrienboer | Courtney Paquette | Fabian Pedregosa | Courtney Paquette | C. Paquette | B. V. Merrienboer
[1] Andrea Montanari,et al. Surprises in High-Dimensional Ridgeless Least Squares Interpolation , 2019, Annals of statistics.
[2] Andrea Montanari,et al. The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve , 2019, Communications on Pure and Applied Mathematics.
[3] R. Durrett. Probability: Theory and Examples , 1993 .
[4] Benjamin Recht,et al. Random Features for Large-Scale Kernel Machines , 2007, NIPS.
[5] J. Demmel. The Probability That a Numerical, Analysis Problem Is Difficult , 2013 .
[6] Walid Hachem,et al. Large complex correlated Wishart matrices: Fluctuations and asymptotic independence at the edges , 2014, 1409.7548.
[7] Arthur Jacot,et al. Neural tangent kernel: convergence and generalization in neural networks (invited paper) , 2018, NeurIPS.
[8] T. Tao,et al. Random Matrices: the Distribution of the Smallest Singular Values , 2009, 0903.0614.
[9] Nathan Srebro,et al. The Marginal Value of Adaptive Gradient Methods in Machine Learning , 2017, NIPS.
[10] Michael J. Todd,et al. Probabilistic Models for Linear Programming , 1991, Math. Oper. Res..
[11] Prateek Jain,et al. Smoothed analysis for low-rank solutions to semidefinite programs in quadratic penalty form , 2018, COLT.
[12] K. Borgwardt. A probabilistic analysis of the simplex method , 1986 .
[13] V. Marčenko,et al. DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES , 1967 .
[14] R. H. Myers,et al. Probability and Statistics for Engineers and Scientists , 1978 .
[15] Zhenyu Liao,et al. A Random Matrix Approach to Neural Networks , 2017, ArXiv.
[16] R. Varga,et al. Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods , 1961 .
[17] Jun Yin,et al. Anisotropic local laws for random matrices , 2014, 1410.3516.
[18] Ruosong Wang,et al. On Exact Computation with an Infinitely Wide Neural Net , 2019, NeurIPS.
[19] Vardan Papyan,et al. The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size. , 2018 .
[20] H. Rutishauser. Theory of Gradient Methods , 1959 .
[21] Marc Teboulle,et al. A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..
[22] J. W. Silverstein,et al. No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices , 1998 .
[23] Thomas Trogdon,et al. Smoothed Analysis for the Conjugate Gradient Algorithm , 2016, 1605.06438.
[24] Shankar Krishnan,et al. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density , 2019, ICML.
[25] Z. Bai,et al. CLT for linear spectral statistics of large dimensional sample covariance matrices with dependent data , 2017, Statistical Papers.
[26] Shang-Hua Teng,et al. Smoothed analysis of algorithms: why the simplex algorithm usually takes polynomial time , 2001, STOC '01.
[27] Zhenyu Liao,et al. The Dynamics of Learning: A Random Matrix Approach , 2018, ICML.
[28] Boris Polyak. Some methods of speeding up the convergence of iteration methods , 1964 .
[29] Surya Ganguli,et al. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization , 2014, NIPS.
[30] J. W. Silverstein,et al. EXACT SEPARATION OF EIGENVALUES OF LARGE DIMENSIONAL SAMPLE COVARIANCE MATRICES , 1999 .
[31] T. Tao. Topics in Random Matrix Theory , 2012 .
[32] Nathan Srebro,et al. Characterizing Implicit Bias in Terms of Optimization Geometry , 2018, ICML.
[33] Michael W. Mahoney,et al. Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning , 2018, J. Mach. Learn. Res..
[34] Thomas Trogdon,et al. Universality in numerical computation with random data. Case studies, analytic results and some speculations , 2016 .
[35] M. Hestenes,et al. Methods of conjugate gradients for solving linear systems , 1952 .
[36] Mert Pilanci,et al. Optimal Randomized First-Order Methods for Least-Squares Problems , 2020, ICML.
[37] Thomas Trogdon,et al. The conjugate gradient algorithm on well-conditioned Wishart matrices is almost deteriministic , 2019, Quarterly of Applied Mathematics.
[38] Francis Bach,et al. On Lazy Training in Differentiable Programming , 2018, NeurIPS.
[39] Thomas Trogdon,et al. Universality in numerical computation with random data: Case studies and analytical results , 2019 .
[40] Jeffrey Pennington,et al. Nonlinear random matrix theory for deep learning , 2019, NIPS.
[41] Jorge Nocedal,et al. Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..
[42] Stephen P. Boyd,et al. A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..
[43] Stephen Smale,et al. On the average number of steps of the simplex method of linear programming , 1983, Math. Program..
[44] J. W. Silverstein,et al. Spectral Analysis of Large Dimensional Random Matrices , 2009 .
[45] Yurii Nesterov,et al. Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.
[46] L. Trefethen,et al. Average-case stability of Gaussian elimination , 1990 .
[47] P. Deift,et al. Universality in numerical computations with random data , 2014, Proceedings of the National Academy of Sciences.
[48] Yann LeCun,et al. Universal halting times in optimization and machine learning , 2015, 1511.06444.
[49] A. Edelman. Eigenvalues and condition numbers of random matrices , 1988 .
[50] Mark W. Schmidt,et al. Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition , 2013, 1308.6370.
[51] Lucas Benigni,et al. Eigenvalue distribution of nonlinear models of random matrices , 2019, ArXiv.
[52] Zeyuan Allen-Zhu,et al. How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD , 2018, NeurIPS.
[53] Teodoro Collin. RANDOM MATRIX THEORY , 2016 .
[54] B. Fischer. Polynomial Based Iteration Methods for Symmetric Linear Systems , 1996 .
[55] Adrien B. Taylor,et al. Smooth strongly convex interpolation and exact worst-case performance of first-order methods , 2015, Mathematical Programming.
[56] P. Deift,et al. How long does it take to compute the eigenvalues of a random, symmetric matrix? , 2012, 1203.4635.
[57] D. A. Flanders,et al. Numerical Determination of Fundamental Modes , 1950 .
[58] Jaehoon Lee,et al. Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes , 2018, ICLR.
[59] Thomas Trogdon,et al. Universality for the Conjugate Gradient and MINRES Algorithms on Sample Covariance Matrices , 2020, Communications on Pure and Applied Mathematics.