Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
暂无分享,去创建一个
[1] J. Schur. Bemerkungen zur Theorie der beschränkten Bilinearformen mit unendlich vielen Veränderlichen. , 1911 .
[2] Yoshua Bengio,et al. Depth with Nonlinearity Creates No Bad Local Minima in ResNets , 2019, Neural Networks.
[3] Michael I. Jordan,et al. Gradient Descent Converges to Minimizers , 2016, ArXiv.
[4] Furong Huang,et al. Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition , 2015, COLT.
[5] Matus Telgarsky,et al. Spectrally-normalized margin bounds for neural networks , 2017, NIPS.
[6] Razvan Pascanu,et al. Revisiting Natural Gradient for Deep Networks , 2013, ICLR.
[7] Ohad Shamir,et al. Size-Independent Sample Complexity of Neural Networks , 2017, COLT.
[8] Matthias Hein,et al. The Loss Surface of Deep and Wide Neural Networks , 2017, ICML.
[9] Surya Ganguli,et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , 2013, ICLR.
[10] Yuandong Tian,et al. An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis , 2017, ICML.
[11] Yann LeCun,et al. Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks , 2018, ArXiv.
[12] James Martens,et al. New Insights and Perspectives on the Natural Gradient Method , 2014, J. Mach. Learn. Res..
[13] Yann LeCun,et al. Improving the convergence of back-propagation learning with second-order methods , 1989 .
[14] Guillaume Hennequin,et al. Exact natural gradient in deep linear networks and its application to the nonlinear case , 2018, NeurIPS.
[15] Ruosong Wang,et al. Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks , 2019, ICML.
[16] Jimmy Ba,et al. Kronecker-factored Curvature Approximations for Recurrent Neural Networks , 2018, ICLR.
[17] Yuanzhi Li,et al. A Convergence Theory for Deep Learning via Over-Parameterization , 2018, ICML.
[18] Wei Hu,et al. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks , 2018, ICLR.
[19] Tengyu Ma,et al. Identity Matters in Deep Learning , 2016, ICLR.
[20] Andrew M. Saxe,et al. High-dimensional dynamics of generalization error in neural networks , 2017, Neural Networks.
[21] Liwei Wang,et al. Gradient Descent Finds Global Minima of Deep Neural Networks , 2018, ICML.
[22] Xiao Zhang,et al. Learning One-hidden-layer ReLU Networks via Gradient Descent , 2018, AISTATS.
[23] Gintare Karolina Dziugaite,et al. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data , 2017, UAI.
[24] Arthur Jacot,et al. Neural tangent kernel: convergence and generalization in neural networks (invited paper) , 2018, NeurIPS.
[25] Elman Mansimov,et al. Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation , 2017, NIPS.
[26] Roger B. Grosse,et al. A Coordinate-Free Construction of Scalable Natural Gradient , 2018, ArXiv.
[27] Roger B. Grosse,et al. Optimizing Neural Networks with Kronecker-factored Approximate Curvature , 2015, ICML.
[28] James Martens,et al. Deep learning via Hessian-free optimization , 2010, ICML.
[29] Yoshua Bengio,et al. Gradient-based learning applied to document recognition , 1998, Proc. IEEE.
[30] Xiaoxia Wu,et al. Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network , 2019, ArXiv.
[31] Pramod Viswanath,et al. Learning One-hidden-layer Neural Networks under General Input Distributions , 2018, AISTATS.
[32] Yann Ollivier,et al. Riemannian metrics for neural networks I: feedforward networks , 2013, 1303.0818.
[33] Yuanzhi Li,et al. Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data , 2018, NeurIPS.
[34] Amir Globerson,et al. Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs , 2017, ICML.
[35] Daniel Soudry,et al. No bad local minima: Data independent training error guarantees for multilayer neural networks , 2016, ArXiv.
[36] Suvrit Sra,et al. Small nonlinearities in activation functions create bad local minima in neural networks , 2018, ICLR.
[37] Jürgen Forster,et al. A linear lower bound on the unbounded error probabilistic communication complexity , 2001, Proceedings 16th Annual IEEE Conference on Computational Complexity.
[38] Ryota Tomioka,et al. Norm-Based Capacity Control in Neural Networks , 2015, COLT.
[39] Nathan Srebro,et al. The Marginal Value of Adaptive Gradient Methods in Machine Learning , 2017, NIPS.
[40] Jaehoon Lee,et al. Wide neural networks of any depth evolve as linear models under gradient descent , 2019, NeurIPS.
[41] James Martens,et al. New perspectives on the natural gradient method , 2014, ArXiv.
[42] Yuan Cao,et al. Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks , 2018, ArXiv.
[43] Peter L. Bartlett,et al. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..
[44] Kenji Kawaguchi,et al. Deep Learning without Poor Local Minima , 2016, NIPS.
[45] Yuanzhi Li,et al. Convergence Analysis of Two-layer Neural Networks with ReLU Activation , 2017, NIPS.
[46] Samy Bengio,et al. Understanding deep learning requires rethinking generalization , 2016, ICLR.
[47] Ronald L. Rivest,et al. Training a 3-node neural network is NP-complete , 1988, COLT '88.
[48] David A. McAllester,et al. A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks , 2017, ICLR.
[49] Samet Oymak,et al. Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks , 2019, IEEE Journal on Selected Areas in Information Theory.
[50] Le Song,et al. Diverse Neural Network Learns True Target Functions , 2016, AISTATS.
[51] Shun-ichi Amari,et al. Neural Learning in Structured Parameter Spaces - Natural Riemannian Gradient , 1996, NIPS.
[52] Guodong Zhang,et al. Noisy Natural Gradient as Variational Inference , 2017, ICML.
[53] Barnabás Póczos,et al. Gradient Descent Provably Optimizes Over-parameterized Neural Networks , 2018, ICLR.
[54] Di He,et al. A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems , 2019, ArXiv.
[55] Gintare Karolina Dziugaite,et al. Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy , 2017, NeurIPS.
[56] David Barber,et al. Practical Gauss-Newton Optimisation for Deep Learning , 2017, ICML.
[57] Yuandong Tian,et al. When is a Convolutional Filter Easy To Learn? , 2017, ICLR.
[58] A. Conv. A Kronecker-factored approximate Fisher matrix for convolution layers , 2016 .
[59] Yi Zhang,et al. Stronger generalization bounds for deep nets via a compression approach , 2018, ICML.
[60] Shun-ichi Amari,et al. Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.