论文信息 - Intrinsic Plasticity via Natural Gradient Descent

Intrinsic Plasticity via Natural Gradient Descent

This paper introduces the natural gradient for intrinsic plas- ticity, which tunes a neuron's activation function such that its output dis- tribution becomes exponentially distributed. The information-geometric properties of the intrinsic plasticity potential are analyzed and the im- proved learning dynamics when using the natural gradient are evaluated for a variety of input distributions. The applied measure for evaluation is the relative geodesic length of the respective path in parameter space.

Klaus Neumann | Jochen J. Steil | J. Steil | Klaus Neumann

[1] Shun-ichi Amari,et al. Differential-geometrical methods in statistics , 1985 .

[2] Geoffrey E. Hinton,et al. Learning representations by back-propagating errors , 1986, Nature.

[3] Geoffrey E. Hinton,et al. Learning representations by back-propagation errors, nature , 1986 .

[4] C. R. Rao,et al. Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[5] Huaiyu Zhu. On Information and Sufficiency , 1997 .

[6] Shun-ichi Amari,et al. Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[7] Shun-ichi Amari,et al. Why natural gradient? , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[8] Jochen Triesch,et al. Synergies between Intrinsic and Synaptic Plasticity in Individual Model Neurons , 2004, NIPS.

[9] P. Lánský,et al. Classification of stationary neuronal activity according to its information rate , 2006, Network.

[10] Jochen J. Steil,et al. Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning , 2007, Neural Networks.