The Contribution of Parameters to Stochastic Complexity

We consider the contribution of parameters to the stochastic complexity. The stochastic complexity of a class of models is the length of a universal, one-part code representing this class. It combines the length of the maximum likelihood code with the parametric complexity, a normalization that acts as a penalty against overfit-ting. For models with few parameters relative to sample size, k n, the parametric complexity is approximately k 2 log n. The accuracy of this approximation, however, deteriorates as k grows relative to n, as occurs in denoising, data mining, and machine learning. For these tasks, the contribution of parameters depends upon the complexity of the model class. Adding a parameter to a model class that already has many produces a different effect than adding one to a model class that has few. In denoising, for example, we show that the parametric complexity leads to an adaptive model selection criterion. We also address the calculation of the para-metric complexity when the underlying integration is unbounded over the natural parameter space, as in Gaussian models.