On the Power of Networks of Majority Functions

Quantization of the synaptic weights is a central problem of hardware implementation of neural networks using numerival technology. In this paper, a particular linear threshold boolean function, called majority function is considered, whose synaptic weights are restricted to only three values: −1, 0, +1. Some results about the complexity of the circuits composed of such gates are reported. They show that this simple family of functions remains powerful in therm of circuit complexity. The learning problem with this subclass of threshold function is also studied and numerical experiments of different algorithms are reported.

[1]  Roman Smolensky,et al.  Algebraic methods in the theory of lower bounds for Boolean circuit complexity , 1987, STOC.

[2]  W. Krauth,et al.  Learning algorithms with optimal stability in neural networks , 1987 .

[3]  W. Krauth,et al.  Storage capacity of memory networks with binary couplings , 1989 .

[4]  Werner Krauth,et al.  Critical storage capacity of the J = ± 1 neural network , 1989 .

[5]  Miklós Ajtai,et al.  ∑11-Formulae on finite structures , 1983, Ann. Pure Appl. Log..

[6]  Fred W. Glover,et al.  Tabu Search - Part I , 1989, INFORMS J. Comput..

[7]  Leslie G. Valiant,et al.  Computational limitations on learning from examples , 1988, JACM.

[8]  A. Razborov.  Lower bounds on the size of bounded depth circuits over a complete basis with logical addition , 1987 .

[9]  Fred Glover,et al.  Tabu Search - Part II , 1989, INFORMS J. Comput..

[10]  Jehoshua Bruck,et al.  On the Power of Threshold Circuits with Small Weights , 1991, SIAM J. Discret. Math..

[11]  Edoardo Amaldi,et al.  Stability-Capacity Diagram of a Neural Network with Ising Bonds , 1989 .

[12]  Michel Verleysen,et al.  Neural networks for high-storage content-addressable memory: VLSI circuit and learning algorithm , 1989 .

[13]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[14]  Johan Håstad,et al.  Almost optimal lower bounds for small depth circuits , 1986, STOC '86.

[15]  Eddy Mayoraz,et al.  Benchmark of Some Learning Algorithms for Single-Layer and Hopfield Networks , 1990, Complex Syst..

[16]  M. A. Epstein.  Lower Bounds for the Size of Circuits of Bounded Depth in Basis F^; G , 1986 .

[17]  Michael Sipser,et al.  Parity, circuits, and the polynomial-time hierarchy , 1981, 22nd Annual Symposium on Foundations of Computer Science (sfcs 1981).

[18]  William H. Kautz,et al.  On the Size of Weights Required for Linear-Input Switching Functions , 1961, IRE Transactions on Electronic Computers.

[19]  Santosh S. Venkatesh,et al.  Directed Drift: A New Linear Threshold Algorithm for Learning Binary Weights On-Line , 1993, J. Comput. Syst. Sci..