Complexity Theory of Neural Networks and Classification Problems

Multilayered, feedforward neural network techniques have been proposed for a variety of classification and recognition problems ranging from speech to sonar signal processing problems. It is generally assumed that the underlying application does not need to be modeled very much and that an artificial neural network solution can be obtained instead by training from empirical data with little or no a priori information about the application. We argue that the right network architecture is fundamental for a good solution to exist and the class of network architectures forms a basis for a complexity theory of classification problems. An abstraction of this notion of complexity leads to ideas similar to Kolmogorov's minimum length description criterion, entropy and k-widths. We will present some basic results on this measure of complexity. From this point of view, artificial neural network solutions to real engineering problems may not ameliorate the difficulties of classification problems, but rather obscure and postpone them. In particular, we doubt that the design of neural networks for solving interesting nontrivial engineering problems will be any easier than other large scale engineering design problems (such as in aerodynamics and semiconductor device modeling).