Statistical Aspects of Generalization in Neural Networks

The topic of this thesis is generalization in feed forward neural networks. The number of parameters in a model and the generalization ability of the model are tightly coupled entities. Neural networks consist usually of a large number of parameters, and the problem of nding the optimal number of parameters in these functions (networks) can be addressed in two levels of `precision': a) as a function of the network degree (the number of internal nodes), or b) as a function of the individual weights (pruning). Both levels are investigated in this thesis, but emphasis is put on pruning seen in the Minimum Descriptive Length (MDL) perspective. There are primarily 2 distinct results: a) Under certain assumptions pruning using a MDL measure can be shown to be equivalent to pruning with Optimal Brain Damage, b) Analysis of the Sunspot series with MDL-pruning shows better generalization than previously [25], and in some cases much better prediction of the future, but this is de nitely not at the point of MDL. This thesis does also contain a introduction to Valiant's Probably Approximately Correct (PAC) learning [26,1,6], Rissanen's Minimum Description Length (MDL) [22,15, 14], and Le Cun's et al. Optimal Brain Damage (OBD) [8,25].