Computational complexity of neural networks: a survey

We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. Our main emphasis is on the computational power of various acyclic and cyclic network models, but we also discuss briefly the complexity aspects of synthesizing networks from examples of their behavior.

[1]  William H. Kautz,et al.  On the Size of Weights Required for Linear-Input Switching Functions , 1961, IRE Transactions on Electronic Computers.

[2]  S. Muroga,et al.  Theory of majority decision elements , 1961 .

[3]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[4]  Saburo Muroga,et al.  Threshold logic and its applications , 1971 .

[5]  Kaoru Nakano,et al.  Associatron-A Model of Associative Memory , 1972, IEEE Trans. Syst. Man Cybern..

[6]  Teuvo Kohonen,et al.  Correlation Matrix Memories , 1972, IEEE Transactions on Computers.

[7]  Teuvo Kohonen,et al.  Representation of Associated Data by Matrix Operators , 1973, IEEE Transactions on Computers.

[8]  Richard J. Lipton,et al.  Some connections between nonuniform and uniform complexity classes , 1980, STOC '80.

[9]  Eric Goles Ch.,et al.  The Convergence of Symmetric Threshold Automata , 1981, Inf. Control..

[10]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[11]  Eric Goles Ch.,et al.  Transient length in sequential iteration of threshold functions , 1983, Discret. Appl. Math..

[12]  Andrew Chi-Chih Yao,et al.  Separating the Polynomial-Time Hierarchy by Oracles (Preliminary Version) , 1985, FOCS.

[13]  Eric Goles Ch.,et al.  Decreasing energy functions as a tool for studying threshold networks , 1985, Discret. Appl. Math..

[14]  Noga Alon.  Asynchronous threshold networks , 1985, Graphs Comb..

[15]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[16]  Johan Håstad,et al.  Almost optimal lower bounds for small depth circuits , 1986, STOC '86.

[17]  L. Personnaz,et al.  Collective computational properties of neural networks: New learning mechanisms. , 1986, Physical review. A, General physics.

[18]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[19]  Ingo Wegener,et al.  The complexity of Boolean functions , 1987 .

[20]  Yves Robert,et al.  Automata networks in computer science : theory and applications , 1987 .

[21]  José L. Balcázar,et al.  On Characterizations of the Class PSPACE/POLY , 1987, Theor. Comput. Sci..

[22]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[23]  Michael Luby,et al.  Steepest Descent Can Take Exponential Time for Symmetric Connection Networks , 1988, Complex Syst..

[24]  James A. Anderson,et al.  Neurocomputing: Foundations of Research , 1988 .

[25]  D. O. Hebb,et al.  The organization of behavior , 1988 .

[26]  Jia-Wei Hong.  On connectionist models , 1988 .

[27]  P. Raghavan,et al.  Learning in threshold networks , 1988, COLT '88.

[28]  J. Stephen Judd,et al.  On the complexity of loading shallow neural networks , 1988, J. Complex..

[29]  Teuvo Kohonen,et al.  Self-organization and associative memory: 3rd edition , 1989 .

[30]  M. Kearns,et al.  Crytographic limitations on learning Boolean formulae and finite automata , 1989, STOC '89.

[31]  Demetri Psaltis,et al.  Linear and logarithmic capacities in associative neural networks , 1989, IEEE Trans. Inf. Theory.

[32]  Pekka Orponen,et al.  On the Computational Complexity of Analyzing Hopfield Nets , 1989, Complex Syst..

[33]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[34]  Anthony Kuh,et al.  Information capacity of associative memories , 1989, IEEE Trans. Inf. Theory.

[35]  Eric Goles Ch.,et al.  Exponential Transient Classes of Symmetric Neural Networks for Synchronous and Sequential Updating , 1989, Complex Syst..

[36]  Amir Dembo,et al.  On the capacity of associative memories with linear threshold functions , 1989, IEEE Trans. Inf. Theory.

[37]  S. Franklin,et al.  Neural computability. II , 1989, International 1989 Joint Conference on Neural Networks.

[38]  Georg Schnitger,et al.  Relating Boltzmann machines to conventional models of computation , 1987, Neural Networks.

[39]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[40]  Zoran Obradovic,et al.  Analog Neural Networks of Limited Precision I: Computing with Multilinear Threshold Functions , 1989, NIPS.

[41]  Martin Hasler,et al.  Recursive neural networks for associative memory , 1990, Wiley-interscience series in systems and optimization.

[42]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[43]  Jehoshua Bruck,et al.  Neural computation of arithmetic functions , 1990 .

[44]  Emile H. L. Aarts,et al.  Simulated annealing and Boltzmann machines - a stochastic approach to combinatorial optimization and neural computing , 1990, Wiley-Interscience series in discrete mathematics and optimization.

[45]  Jirí Wiedermann,et al.  Complexity Issues in Discrete Neurocomputing , 1990, IMYCS.

[46]  Jehoshua Bruck.  On the convergence properties of the Hopfield model , 1990, Proc. IEEE.

[47]  Max H. Garzon,et al.  Global Dynamics in Neural Networks II , 1990, Complex Syst..

[48]  J. Stephen Judd,et al.  Neural network design and the complexity of learning , 1990, Neural network modeling and connectionism.

[49]  J.A. Anderson,et al.  Directions for research , 1990 .

[50]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[51]  Mihalis Yannakakis,et al.  Simple Local Search Problems That are Hard to Solve , 1991, SIAM J. Comput..

[52]  Alon Orlitsky,et al.  A geometric approach to threshold circuit complexity , 1991, COLT '91.

[53]  Georg Schnitger,et al.  On the computational power of sigmoid versus Boolean threshold circuits , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[54]  Jehoshua Bruck,et al.  On the Power of Threshold Circuits with Small Weights , 1991, SIAM J. Discret. Math..

[55]  Hava T. Siegelmann,et al.  On the computational power of neural nets , 1992, COLT '92.

[56]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[57]  Kai-Yeung Siu,et al.  Optimal Depth Neural Networks for Multiplication and Related Problems , 1992, NIPS.

[58]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[59]  J. Reif,et al.  On Threshold Circuits and Polynomial Computation , 1992, SIAM J. Comput..

[60]  Georg Schnitger,et al.  The Power of Approximation: A Comparison of Activation Functions , 1992, NIPS.

[61]  Eduardo D. Sontag,et al.  Feedforward Nets for Interpolation and Classification , 1992, J. Comput. Syst. Sci..

[62]  Marek Karpinski,et al.  Simulating threshold circuits by majority circuits , 1993, SIAM J. Comput..

[63]  Alexander A. Razborov,et al.  n^Omega(log n) Lower Bounds on the Size of Depth-3 Threshold Circuits with AND Gates at the Bottom , 1993, Information Processing Letters.

[64]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[65]  Pekka Orponen,et al.  On the Computational Power of Discrete Hopfield Nets , 1993, ICALP.

[66]  Wolfgang Maass,et al.  Bounds for the computational power and learning complexity of analog neural nets , 1993, SIAM J. Comput..

[67]  Pekka Orponen,et al.  Attraction Radii in Binary Hopfield Nets are Hard to Compute , 1993, Neural Computation.

[68]  Ian Parberry,et al.  Circuit complexity and neural networks , 1994 .

[69]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[70]  Johan Håstad,et al.  On the Size of Weights for Threshold Gates , 1994, SIAM J. Discret. Math..

[71]  José L. Balcázar,et al.  Structural Complexity I , 1995, Texts in Theoretical Computer Science An EATCS Series.