Dynamics of Gradient-Based Learning and Applications to Hyperparameter Estimation

We analyse the dynamics of gradient-based learning algorithms using the cavity method, considering the cases of batch learning with non-vanishing rates, and on-line learning. It has an an excellent agreement with simulations. Applications to efficient and precise estimation of hyperparameters are proposed.

[1]  F. Li,et al.  Fast Parameter Estimation Using Green's Functions , 2001, NIPS.

[2]  Tong,et al.  Many-body approach to the dynamics of batch learning , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[3]  Klaus-Robert Müller,et al.  Asymptotic statistical theory of overtraining and cross-validation , 1997, IEEE Trans. Neural Networks.

[4]  Peixun Luo,et al.  Dynamical and stationary properties of on-line learning from finite training sets. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[5]  David Saad,et al.  On-Line Learning in Neural Networks , 1999 .

[6]  M. Opper,et al.  Dynamics of batch training in a perceptron , 1998 .

[7]  A. Coolen,et al.  Supervised learning with restricted training sets: a generating functional analysis , 2001 .

[8]  David Barber,et al.  On-line learning from finite training sets , 1997 .