Some notes on the stability of learning

Learning theory based on ERM principle, especially promoted by VC theory provides some conditions on the hypothesis space to ensure generalization. However, several successful learning algorithms including regularization learning, SVM, bagging and boost are not strictly ERM. So, scientists are looking for new foundation of learning. Stability conditions are perhaps new foundation. We give an exponential bound for generalization performance based on concentration inequality with strong CV stability.

[1]  T. Poggio,et al.  General conditions for predictivity in learning theory , 2004, Nature.

[2]  S. Kutin.  Extensions to McDiarmid's inequality when dierences are bounded with high probability , 2002 .

[3]  Noga Alon,et al.  Scale-sensitive dimensions, uniform convergence, and learnability , 1997, JACM.

[4]  Dana Ron,et al.  Algorithmic Stability and Sanity-Check Bounds for Leave-one-Out Cross-Validation , 1997, COLT.

[5]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[6]  Luc Devroye,et al.  Distribution-free performance bounds for potential function rules , 1979, IEEE Trans. Inf. Theory.

[7]  Robert E. Schapire,et al.  A Brief Introduction to Boosting , 1999, IJCAI.

[8]  André Elisseeff,et al.  Stability and Generalization , 2002, J. Mach. Learn. Res..

[9]  S. Mendelson.  Geometric Parameters in Learning Theory , 2004 .

[10]  T. Poggio,et al.  On Stability and Concentration of Measure , 2004 .

[11]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[12]  Colin McDiarmid,et al.  Surveys in Combinatorics, 1989: On the method of bounded differences , 1989 .

[13]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[14]  László Györfi,et al.  A Probabilistic Theory of Pattern Recognition , 1996, Stochastic Modelling and Applied Probability.

[15]  Paul Van Dooren,et al.  Stability margins for generalized state space systems , 2006, Appl. Math. Lett..

[16]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .