Programmed interactions in higher-order neural networks: Maximal capacity

Abstract The focus of the paper is the estimation of the maximum number of states that can be made stable in higher-order extensions of neural network models. Each higher-order neuron in a network of n elements is modeled as a polynomial threshold element of degree d . It is shown that regardless of the manner of operation, or the algorithm used, the storage capacity of the higher-order network is of the order of one bit per interaction weight. In particular, the maximal (algorithm independent) storage capacity realizable in a recurrent network of n higher-order neurons of degree d is of the order of n d d! . A generalization of a spectral algorithm for information storage is introduced and arguments adducing near optimal capacity for the algorithm are presented.

[1]  B V Kumar,et al.  Evaluation of the use of the Hopfield neural network model as a nearest-neighbor algorithm. , 1986, Applied optics.

[2]  I. Guyon,et al.  Information storage and retrieval in spin-glass like neural networks , 1985 .

[3]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[4]  Demetri Psaltis,et al.  Linear and logarithmic capacities in associative neural networks , 1989, IEEE Trans. Inf. Theory.

[5]  Pierre Baldi,et al.  On Properties of Networks of Neuron-Like Elements , 1987, NIPS.

[6]  S. Venkatesh.  Linear Maps with Point Rules: Applications to Pattern Classification and Associative Memory , 1987 .

[7]  Demetri Psaltis,et al.  Nonlinear discriminant functions and associative memories , 1987 .

[8]  Zoltán Füredi.  Random polytopes in thed-dimensional cube , 1986, Discret. Comput. Geom..

[9]  Baldi,et al.  Number of stable points for spin-glasses and neural networks of higher orders. , 1987, Physical review letters.

[10]  Santosh S. Venkatesh,et al.  Programmed interactions in higher-order neural networks: The outer-product algorithm , 1991, J. Complex..

[11]  Eric Goles,et al.  Lyapunov function for parallel neural networks , 1987 .

[12]  Yaser S. Abu-Mostafa,et al.  Information capacity of the Hopfield model , 1985, IEEE Trans. Inf. Theory.

[13]  Pierre Baldi,et al.  Random interactions in higher order neural networks , 1993, IEEE Trans. Inf. Theory.

[14]  C. L. Giles,et al.  Machine learning using higher order correlation networks , 1986 .

[15]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[16]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[17]  H. Schwarz.  Gesammelte mathematische Abhandlungen , 1970 .

[18]  Pierre Baldi,et al.  Neural networks, orientations of the hypercube, and algebraic threshold functions , 1988, IEEE Trans. Inf. Theory.

[19]  Demetri Psaltis,et al.  On Reliable Computation With Formal Neurons , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  J. G. Wendel.  A Problem in Geometric Probability. , 1962 .

[21]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..