Deeper Sparsely Nets can be Optimal

The starting points of this paper are two size-optimal solutions: (i) one for implementing arbitrary Boolean functions [1]; and (ii) another one for implementing certain sub-classes of Boolean functions [2]. Because VLSI implementations do not cope well with highly interconnected nets – the area of a chip grows with the cube of the fan-in [3] – this paper will analyse the influence of limited fan-in on the size optimality for the two solutions mentioned. First, we will extend a result from Horne & Hush [1] valid for fan-ins Δ = 2 to arbitrary fan-in. Second, we will prove that size-optimal solutions are obtained for small constant fan-in for both constructions, while relative minimum size solutions can be obtained for fan-ins strictly lower than linear. These results are in agreement with similar ones proving that for small constant fan-ins (Δ = 6 ... 9), there exist VLSI-optimal (i.e., minimising AT2) solutions [4], while there are similar small constants relating to our capacity of processing information [5].

[1]  Don R. Hush,et al.  On the node complexity of neural networks , 1994, Neural Networks.

[2]  Dan Hammerstrom,et al.  The Connectivity Analysis of Simple Association - or- How Many Connections Do You Need! , 1988 .

[3]  Robert C. Williamson.  e-Entropy and the Complexity of Feedforward Neural Networks , 1990, NIPS 1990.

[4]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[5]  Thomas Kailath,et al.  Depth-Size Tradeoffs for Neural Computation , 1991, IEEE Trans. Computers.

[6]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[7]  N. P. Red’kin.  Synthesis of threshold circuits for certain classes of Boolean functions , 1970 .

[8]  Lex A. Akers,et al.  Training a Limited-Interconnect, Synthetic Neural IC , 1988, NIPS.

[9]  Valeriu Beiu,et al.  On the Circuit Complexity of Sigmoid Feedforward Neural Networks , 1996, Neural Networks.

[10]  Alon Orlitsky,et al.  Lower bounds on threshold and related circuits via communication complexity , 1994, IEEE Trans. Inf. Theory.

[11]  Dhananjay S. Phatak,et al.  Connectivity and performance tradeoffs in the cascade correlation learning architecture , 1994, IEEE Trans. Neural Networks.

[12]  Valeriu Beiu,et al.  When constants are important , 1997 .

[13]  Gary G. R. Green,et al.  Neural networks, approximation theory, and finite precision computation , 1995, Neural Networks.

[14]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[15]  Valeriu Beiu,et al.  On the circuit and VLSI complexity of threshold gate COMPARISON , 1998, Neurocomputing.

[16]  Joseph W. Goodman,et al.  On the power of neural networks for solving hard problems , 1990, J. Complex..

[17]  Yaser S. Abu-Mostafa,et al.  Connectivity Versus Entropy , 1987, NIPS.

[18]  Joos Vandewalle,et al.  Area-Time Performances of Some Neural Computations , 1994 .

[19]  Stamatis Vassiliadis,et al.  2-1 Additions and Related Arithmetic Operations with Threshold Logic , 1996, IEEE Trans. Computers.

[20]  Byoung-Tak Zhang,et al.  Genetic Programming of Minimal Neural Nets Using Occam's Razor , 1993, ICGA.

[21]  V. Beiu,et al.  Constant fan-in digital neural networks are VLSI-optimal , 1995 .