Measuring generalization quality

The concept of measure functions for generalization performance is suggested. This concept provides an alternative way of designing and selecting generalization algorithms. In addition, it makes a clear distinction between modeling and solving a computational problem. The modeling is captured in a measure function that, for each possible combination of a training set and a generalization, assigns a value describing how good the generalization is. The computational problem is to find a generalization maximizing the measure function. With this concept in use, some recently debated facts about quality of generalization will become clarified. In addition to their theoretical relevance, we argue that measure functions are of great value for practical applications. For instance, (i) they force us to make explicit the relevant features of the generalization problem at hand, (ii) they provide a deeper understanding of existing generalization algorithms, and (iii) they help us in the construction of problem-specific algorithms. We illustrate the second point by an experiment that indicates that the difference between generalizations computed by different algorithms is often smaller than the difference between the generalizations computed by different versions of the same algorithm. The third point is supported by a novel algorithm based on incremental search for a generalization that optimizes a given measure function.