论文信息 - Asymptotic Convergence of Backpropagation: Numerical Experiments

Asymptotic Convergence of Backpropagation: Numerical Experiments

We have calculated, both analytically and in simulations, the rate of convergence at long times in the backpropagation learning algorithm for networks with and without hidden units. Our basic finding for units using the standard sigmoid transfer function is 1/t convergence of the error for large t, with at most logarithmic corrections for networks with hidden units. Other transfer functions may lead to a slower polynomial rate of convergence. Our analytic calculations were presented in (Tesauro, He & Ahamd, 1989). Here we focus in more detail on our empirical measurements of the convergence rate in numerical simulations, which confirm our analytic results.

[1] Geoffrey E. Hinton,et al. Learning representations by back-propagating errors , 1986, Nature.

[2] Gerald Tesauro,et al. A study of scaling and generalization in neural networks , 1988, Neural Networks.

[3] Robert A. Jacobs,et al. Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[4] Gerald Tesauro,et al. Scaling and Generalization in Neural Networks: A Case Study , 1988, NIPS.

[5] Yu He,et al. Asymptotic Convergence of Backpropagation , 1989, Neural Computation.

[6] Geoffrey E. Hinton. Connectionist Learning Procedures , 1989, Artif. Intell..