Divide and Conquer Neural Networks

Determining an effective architecture far a multi-layer feedforward back propagation neural network can be a time-consuming effort. We describe an algorithm called Divide and Conquer Neural Networks (DCN), which creates a feedforward neural network architecture during training, based upon the training examples. The first cell introduced on any layer is trained on all examples. Further cells on a layer are trained primarily on examples not already correctly classified. The learning algorithm is shown to be able to use several different learning rules, including the delta rule and perceptron rule, to modify the link weights one level at a time in the spirit of a perceptron. Error is never propagated backwards through a hidden cell. Examples are shown of networks generated for the exdusive-or, 4 and 5-parity, 2-spirals problem, Iris plant classification, predicting party affiliation from voting records, and the real-valued fuzzy exclusive-or. The results show the algorithm effectively learns viable architectures that can generalize.

[1]  Lawrence O. Hall,et al.  A Hybrid Connectionist, Symbolic Learning System , 1990, AAAI.

[2]  Tony R. Martinez,et al.  A Self-Adjusting Dynamic Logic Module , 1991, J. Parallel Distributed Comput..

[3]  Michael C. Mozer,et al.  Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment , 1988, NIPS.

[4]  Charles Leave.  Neural Networks: Algorithms, Applications and Programming Techniques , 1992 .

[5]  Sholom M. Weiss,et al.  An Empirical Comparison of Pattern Recognition, Neural Nets, and Machine Learning Classification Methods , 1989, IJCAI.

[6]  Lawrence O. Hall,et al.  Fuzzy quantifiers and quantifying operators in a connectionist expert system development tool , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[7]  Michael de la Maza.  A Prototype Based Symbolic Concept Learning System , 1991, ML.

[8]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[9]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[10]  Terence D. Sanger,et al.  A tree-structured adaptive network for function approximation in high-dimensional spaces , 1991, IEEE Trans. Neural Networks.

[11]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[12]  Dennis Connolly,et al.  Self organizing modular neural networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[13]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[14]  S. G. Romaniuk,et al.  Fuzzy connectionist expert systems , 1989, International 1989 Joint Conference on Neural Networks.

[15]  Philip D. Wasserman,et al.  Neural computing - theory and practice , 1989 .

[16]  Alberto M. Segre.  Computer systems that learn: Sholom M. Weiss and Casimir A. Kulikowski☆ , 1993 .

[17]  Yamashita,et al.  Backpropagation algorithm which varies the number of hidden units , 1989 .

[18]  John M. Zelle,et al.  Growing layers of perceptrons: introducing the Extentron algorithm , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[19]  J. C. Schlimmer,et al.  Concept acquisition through representational adjustment , 1987 .