Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
暂无分享,去创建一个
Surya Ganguli | Samuel S. Schoenholz | Jeffrey Pennington | Jeffrey Pennington | S. Ganguli | S. Schoenholz
[1] R. Speicher. Multiplicative functions on the lattice of non-crossing partitions and free convolution , 1994 .
[2] O. Johnson. Free Random Variables , 2004 .
[3] Geoffrey E. Hinton,et al. Reducing the Dimensionality of Data with Neural Networks , 2006, Science.
[4] Yoshua Bengio,et al. Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.
[5] T. Tao. Topics in Random Matrix Theory , 2012 .
[6] Geoffrey E. Hinton,et al. ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.
[7] Thorsten Neuschel. Plancherel-Rotach formulae for average characteristic polynomials of products of Ginibre random matrices and the Fuss-Catalan distribution , 2013, 1311.0365.
[8] Razvan Pascanu,et al. On the difficulty of training recurrent neural networks , 2012, ICML.
[9] Ha Hong,et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.
[10] Surya Ganguli,et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , 2013, ICLR.
[11] Jimmy Ba,et al. Adam: A Method for Stochastic Optimization , 2014, ICLR.
[12] Leonidas J. Guibas,et al. Deep Knowledge Tracing , 2015, NIPS.
[13] Surya Ganguli,et al. Deep Learning Models of the Retinal Response to Natural Scenes , 2017, NIPS.
[14] Surya Ganguli,et al. Exponential expressivity in deep neural networks through transient chaos , 2016, NIPS.
[15] Demis Hassabis,et al. Mastering the game of Go with deep neural networks and tree search , 2016, Nature.
[16] Jiri Matas,et al. All you need is a good init , 2015, ICLR.
[17] George Kurian,et al. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , 2016, ArXiv.
[18] Surya Ganguli,et al. Deep Information Propagation , 2016, ICLR.
[19] Quoc V. Le,et al. Neural Architecture Search with Reinforcement Learning , 2016, ICLR.