Further Readings

tions will be similarly accurate. For example, a large network with many parameters may be capable of achieving a small error on the training set, and yet fail to model the underlying distribution of the data and hence achieve poor performance on new data (a phenomenon sometimes called “overfitting”). This problem can be approached by limiting the complexity of the model, thereby forcing it to extract regularities in the data rather than simply memorizing the training set. From a fully probabilistic viewpoint, learning in feedforward networks involves using the network to define a prior distribution over functions, which is converted to a posterior distribution once the training data have been observed. It can be formalized through the framework of BAYESIAN LEARNING, or equivalently through the MINIMUM DESCRIPTION LENGTH approach (MacKay 1992; Neal 1996). In practical applications of feedforward networks, attention must be paid to the representation used for the data. For example, it is common to perform some kind of preprocessing on the raw input data (perhaps in the form of “feature extraction”) before they are used as inputs to the network. Often this preprocessing takes into consideration any prior knowledge we might have about the desired properties of the solution. For instance, in the case of digit recognition we know that the identity of the digit should be invariant to the position of the digit within the input image. Feedforward neural networks are now well established as an important technique for solving pattern recognition problems, and indeed there are already many commercial applications of feedforward neural networks in routine use. See also COGNITIVE MODELING, CONNECTIONIST; CONNECTIONIST APPROACHES TO LANGUAGE; MCCULLOCH; NEURAL NETWORKS; PITTS; RECURRENT NETWORKS