On the complexity of neural networks with sigmoidal units

Novel techniques based on classical tools such as rational approximation and harmonic analysis are developed to study the computational properties of neural networks. Using such techniques, one can characterize the class of function whose complexity is almost the same among various models of neural networks with feedforward structures. As a consequence of this characterization, for example, it is proved that any depth-(d+1) network of sigmoidal units computing the parity function of n inputs must have Omega (dn/sup 1/d- in /) units, for any fixed in >0. This lower bound is almost tight since one can compute the parity function with O(dn/sup 1/d/) sigmoidal units in a depth-(d+1) network. The techniques also generalize to networks whose elements can be approximated by piecewise low degree rational functions.<<ETX>>

[1]  T. Kailath,et al.  Computing With Almost Optimal Size Threshold Circuits , 1991, Proceedings. 1991 IEEE International Symposium on Information Theory.

[2]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[3]  K. Siu.  Complexity issues in circuits for neural computation , 1992 .

[4]  Jehoshua Bruck,et al.  Depth efficient neural networks for division and related problems , 1993, IEEE Trans. Inf. Theory.

[5]  P. Petrushev,et al.  Rational Approximation of Real Functions , 1988 .

[6]  Thomas Kailath,et al.  Depth-Size Tradeoffs for Neural Computation , 1991, IEEE Trans. Computers.

[7]  Georg Schnitger,et al.  On the computational power of sigmoid versus Boolean threshold circuits , 1991, [1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science.

[8]  Jehoshua Bruck,et al.  Neural computation of arithmetic functions , 1990 .

[9]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[10]  A. Gončar.  ON THE RAPIDITY OF RATIONAL APPROXIMATION OF CONTINUOUS FUNCTIONS WITH CHARACTERISTIC SINGULARITIES , 1967 .

[11]  Alon Orlitsky,et al.  A geometric approach to threshold circuit complexity , 1991, COLT '91.

[12]  Michael E. Saks,et al.  On threshold circuits for parity , 1990, Proceedings [1990] 31st Annual Symposium on Foundations of Computer Science.

[13]  Robert J. Lechner.  HARMONIC ANALYSIS OF SWITCHING FUNCTIONS , 1971 .

[14]  Pavel Pudlák,et al.  Threshold circuits of bounded depth , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[15]  Jehoshua Bruck,et al.  On the Power of Threshold Circuits with Small Weights , 1991, SIAM J. Discret. Math..