Constructive Training Methods for feedforward Neural Networks with Binary weights

Quantization of the parameters of a Perceptron is a central problem in hardware implementation of neural networks using a numerical technology. A neural model with each weight limited to a small integer range will require little surface of silicon. Moreover, according to Occam's razor principle, better generalization abilities can be expected from a simpler computational model. The price to pay for these benefits lies in the difficulty to train these kind of networks. This paper proposes essentially two new ideas for constructive training algorithms, and demonstrates their efficiency for the generation of feedforward networks composed of Boolean threshold gates with discrete weights. A proof of the convergence of these algorithms is given. Some numerical experiments have been carried out and the results are presented in terms of the size of the generated networks and of their generalization abilities.

[1]  Jean-Pierre Nadal,et al.  Neural trees: a new tool for classification , 1990 .

[2]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[3]  Eddy Mayoraz,et al.  On the Power of Democratic Networks , 1996, SIAM J. Discret. Math..

[4]  Jean-Pierre Nadal,et al.  Study of a Growth Algorithm for a Feedforward Network , 1989, Int. J. Neural Syst..

[5]  J. L. Holt,et al.  Back propagation simulations using limited precision calculations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[6]  M. Golea,et al.  A Growth Algorithm for Neural Network Decision Trees , 1990 .

[7]  Florence d'Alché-Buc,et al.  Trio Learning: A New Strategy for Building Hybrid Neural Trees , 1994, Int. J. Neural Syst..

[8]  E. Fiesler,et al.  Comparative Bibliography of Ontogenic Neural Networks , 1994 .

[9]  D. Yeung,et al.  Constructive feedforward neural networks for regression problems : a survey , 1995 .

[10]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[11]  R.J.F. Dow,et al.  Neural net pruning-why and how , 1988, IEEE 1988 International Conference on Neural Networks.

[12]  Guillaume Deffuant,et al.  An Algorithm for Building Regularized Piecewise Linear Discrimination Surfaces: The Perceptron Membrane , 1995, Neural Computation.

[13]  Eddy Mayoraz,et al.  Maximizing the robustness of a linear threshold classifier with discrete weights , 1994 .

[14]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[15]  John J. Paulos,et al.  The Effects of Precision Constraints in a Backpropagation Learning Network , 1990, Neural Computation.

[16]  Eddy Mayoraz,et al.  On the Power of Networks of Majority Functions , 1991, IWANN.

[17]  Juan-Manuel Torres-Moreno,et al.  An evolutive architecture coupled with optimal perceptron learning for classification , 1995, ESANN.

[18]  K. Asanovi.  Experimental Determination of Precision Requirements for Back-propagation Training of Artiicial Neural Networks , 1991 .

[19]  Edoardo Amaldi,et al.  Two Constructive Methods for Designing Compact Feedforward Networks of Threshold Units , 1997, Int. J. Neural Syst..

[20]  Y. Chien,et al.  Pattern classification and scene analysis , 1974 .

[21]  Jehoshua Bruck,et al.  On the Power of Threshold Circuits with Small Weights , 1991, SIAM J. Discret. Math..

[22]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[23]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[24]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[25]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[26]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[27]  G. Barkema,et al.  A Fast Partitioning Algorithm and a Comparison of Binary Feedforward Neural Networks , 1992 .

[28]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[29]  Jenq-Neng Hwang,et al.  Finite Precision Error Analysis of Neural Network Hardware Implementations , 1993, IEEE Trans. Computers.

[30]  Eddy Mayoraz,et al.  A constructive training algorithm for feedforward neural networks with ternary weights , 1994, ESANN.

[31]  Noga Alon,et al.  Explicit Constructions of Depth-2 Majority Circuits for Comparison and Addition , 1994, SIAM J. Discret. Math..