Constructing Higher Order Neurons of Increasing Complexity in Cascade Networks

A problem faced by many constructive neural networks using a cascade architecture is the large network depth. This results in large fan-in and propagation delays, problems especially relevant for VLSI implementation of these networks. This work explores the effect of limiting the depth of the cascades created by CasPer, a constructive cascade algorithm. Instead of a single cascade of hidden neurons, a series of cascade towers are built. Each cascade tower can be viewed as a single Higher Order Neuron (HON). The optimal complexity of the HON required for a given problem is difficult to estimate, and is a form of the bias-variance dilemma. This problem is overcome via the construction of HONs with increasing complexity. It is shown that by constructing HONs in this manner the chance of overfitting is reduced, especially with noisy data.