Exploring the Capacity of Simple Neural Networks

The application of neural networks in pattern recognition problems constitute some difficult problems like initialization, choosing the correct architecture etc. In this paper the problem of ”how do networks find non-linear solutions” is addressed. By examining the properties of a simple (one input and one output) multi layered perceptron with sigmoidal transfer functions we try to show in what extend the network is able to find non-linear solutions starting in a linear initialization. In order to formalize the network’s behaviour, the discrimination capacity DC is introduced. This measure indicates whether the network has found any non-linear solutions. Experiments show that the values for DC that can be expected from the theory are indeed met but the mathematical analysis of the network is too complicated to exactly point out the exact points where specific changes of DC can be found.