On Stochastic Complexity and Admissible Models for Neural Network Classifiers

Given some training data how should we choose a particular network classifier from a family of networks of different complexities? In this paper we discuss how the application of stochastic complexity theory to classifier design problems can provide some insights into this problem. In particular we introduce the notion of admissible models whereby the complexity of models under consideration is affected by (among other factors) the class entropy, the amount of training data, and our prior belief. In particular we discuss the implications of these results with respect to neural architectures and demonstrate the approach on real data from a medical diagnosis task.