Generalization, surface-fitting, and network structures

Most overtly considered in the field of neural nets, the problem of how best to generalize from a given learning set of input-output examples is also central to the fields of statistics, approximation theory, and artificial intelligence. This dissertation investigates this problem from within an abstract framework and then tests some of the resultant concepts in real-world situations. In this abstract framework a generalizer is completely specified by a certain countably infinite set of functions, so the mathematics of generalization becomes an investigation into candidate sets of criteria governing the behavior of that infinite set of functions. One such set of criteria defines the hyperplanar HERBIE, a generalizer based on surface-fitting procedures rather than neural networks. It is argued that such a HERBIE is well-suited to serving as a benchmark of generalization efficacy, and its real-world generalizing is then favorably compared to neural nets for several real generalization problems, including Sejnowski's problem of reading aloud. This leads to the conclusion that (current) neural nets in fact constitute a poor means of generalizing. Other, more sophisticated sets of criteria can readily be approximated as computer programs, some of which employ network structures built via an evolutionary process. A preliminary and favorable investigation into the behavior of these approximations finishes this dissertation.