Using random weights to train multilayer networks of hard-limiting units

A gradient descent algorithm suitable for training multilayer feedforward networks of processing units with hard-limiting output functions is presented. The conventional backpropagation algorithm cannot be applied in this case because the required derivatives are not available. However, if the network weights are random variables with smooth distribution functions, the probability of a hard-limiting unit taking one of its two possible values is a continuously differentiable function. In the paper, this is used to develop an algorithm similar to backpropagation, but for the hard-limiting case. It is shown that the computational framework of this algorithm is similar to standard backpropagation, but there is an additional computational expense involved in the estimation of gradients. Upper bounds on this estimation penalty are given. Two examples which indicate that, when this algorithm is used to train networks of hard-limiting units, its performance is similar to that of conventional backpropagation applied to networks of units with sigmoidal characteristics are presented.

[1]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[2]  Bernard Widrow,et al.  Neural nets for adaptive filtering and adaptive pattern recognition , 1988, Computer.

[3]  D. J. Toms,et al.  Training binary node feedforward neural networks by back propagation of error , 1990 .

[4]  Francis Crick,et al.  The recent excitement about neural networks , 1989, Nature.

[5]  Sammy Siu,et al.  Multilayer perceptron structures applied to adaptive equalisers for data communications , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[6]  Geoffrey E. Hinton.  20 – CONNECTIONIST LEARNING PROCEDURES1 , 1990 .

[7]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[8]  David Haussler,et al.  Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications , 1992, Inf. Comput..

[9]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[10]  Nicholas J. Redding,et al.  Learning in Feedforward Networks with Nonsmooth Functions , 1991, NIPS.

[11]  Stephen Jose Hanson,et al.  Meiosis Networks , 1989, NIPS.

[12]  D. Pollard.  Convergence of stochastic processes , 1984 .

[13]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[14]  Esther Levin,et al.  Accelerated Learning in Layered Neural Networks , 1988, Complex Syst..

[15]  J. Peng,et al.  Reinforcement learning algorithms as function optimizers , 1989, International 1989 Joint Conference on Neural Networks.

[16]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.