Neural Computation with Winner-Take-All as the Only Nonlinear Operation

Everybody "knows" that neural networks need more than a single layer of nonlinear units to compute interesting functions. We show that this is false if one employs winner-take-all as nonlinear unit: • Any boolean function can be computed by a single k-winner-take-all unit applied to weighted sums of the input variables. • Any continuous function can be approximated arbitrarily well by a single soft winner-take-all unit applied to weighted sums of the input variables. • Only positive weights are needed in these (linear) weighted sums. This may be of interest from the point of view of neurophysiology, since only 15% of the synapses in the cortex are inhibitory. In addition it is widely believed that there are special microcircuits in the cortex that compute winner-take-all. • Our results support the view that winner-take-all is a very useful basic computational unit in Neural VLSI: □ it is wellknown that winner-take-all of n input variables can be computed very efficiently with 2n transistors (and a total wire length and area that is linear in n) in analog VLSI [Lazzaro et at., 1989] □ we show that winner-take-all is not just useful for special purpose computations, but may serve as the only nonlinear unit for neural circuits with universal computational power □ we show that any multi-layer perceptron needs quadratically in n many gates to compute winner-take-all for n input variables, hence winner-take-all provides a substantially more powerful computational unit than a perceptron (at about the same cost of implementation in analog VLSI). Complete proofs and further details to these results can be found in [Maass, 2000].