Performance analysis of a pipelined backpropagation parallel algorithm

The supervised training of feedforward neural networks is often based on the error backpropagation algorithm. The authors consider the successive layers of a feedforward neural network as the stages of a pipeline which is used to improve the efficiency of the parallel algorithm. A simple placement rule is used to take advantage of simultaneous executions of the calculations on each layer of the network. The analytic expressions show that the parallelization is efficient. Moreover, they indicate that the performance of this implementation is almost independent of the neural network architecture. Their simplicity assures easy prediction of learning performance on a parallel machine for any neural network architecture. The experimental results are in agreement with analytical estimates.

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Inmos Limited,et al.  OCCAM 2 reference manual , 1988 .

[3]  C. Cowan,et al.  Adaptive Filters and Equalisers , 1988 .

[4]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.

[5]  Michel Weinfeld,et al.  A Fully Digital Integrated CMOS Hopfield Network Including the Learning Algorithm , 1989 .

[6]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[7]  Etienne Deprit.  Implementing recurrent back-propagation on the connection machine , 1989, Neural Networks.

[8]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[9]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[10]  José A. B. Fortes,et al.  Performance of Connectionist Learning Algorithms on 2-D SIMD Processor Arrays , 1989, NIPS.

[11]  Alexander Singer.  Exploiting the Inherent Parallelism of Artificial Neural Networks to Achieve 1300 Million Interconnects per Second , 1990 .

[12]  Alexander Singer,et al.  Implementations of artificial neural networks on the Connection Machine , 1990, Parallel Comput..

[13]  J. C. Mignot,et al.  Implementations of multilayer neural networks on parallel architectures , 1991 .