Theoretical analysis of a class of randomized regularization methods

The convergence behavior o-f traditional leaming algorithms can be analyzed in the VC theoretical framework. Recently, many researchers have been interested in a class of randomized learning algorithms including the Gibbs algorithm from statistical mechanics. However, no successful theory concerning the generalization behavior of these randomized learning algorithms have been established previously. In order to fully understand the behavior of these randomized estimators, we shall compare them with regularization schemes for deterministic estimators. Furthermore, we present theoretical analysis for such algorithms which leads to rigorous convergence bounds.

[1]  David Haussler,et al.  Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension , 1991, COLT '91.

[2]  Leslie G. Valiant,et al.  Fast probabilistic algorithms for hamiltonian circuits and matchings , 1977, STOC '77.

[3]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .

[4]  J. Rissanen.  Stochastic Complexity in Statistical Inquiry Theory , 1989 .

[5]  Opper,et al.  Bounds for predictive errors in the statistical mechanics of supervised learning. , 1995, Physical review letters.

[6]  D. Haussler,et al.  Rigorous learning curve bounds from statistical mechanics , 1996 .

[7]  David Haussler,et al.  Rigorous Learning Curve Bounds from Statistical Mechanics , 1994, COLT.

[8]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[9]  中澤 真,et al.  Devroye, L., Gyorfi, L. and Lugosi, G. : A Probabilistic Theory of Pattern Recognition, Springer (1996). , 1997 .

[10]  Tatsuya Uezu.  Department of Physics, Nara Women's University, Nara 630, Japan: Learning from stochastic rules under finite temperature - optimal temperature and asymptotic learning curve , 1997 .

[11]  A. Krogh,et al.  Statistical mechanics of ensemble learning , 1997 .

[12]  P. Gänssler.  Weak Convergence and Empirical Processes - A. W. van der Vaart; J. A. Wellner. , 1997 .

[13]  Sompolinsky,et al.  Statistical mechanics of learning from examples. , 1992, Physical review. A, Atomic, molecular, and optical physics.

[14]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[15]  T. Watkin,et al.  THE STATISTICAL-MECHANICS OF LEARNING A RULE , 1993 .

[16]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[17]  David A. McAllester.  Some PAC-Bayesian theorems , 1998, COLT' 98.