Efficient mapping of backpropagation algorithm onto a network of workstations

In this paper, we present an efficient technique for mapping a backpropagation (BP) learning algorithm for multilayered neural networks onto a network of workstations (NOW's). We adopt a vertical partitioning scheme, where each layer in the neural network is divided into p disjoint partitions, and map each partition onto an independent workstation in a network of p workstations. We present a fully distributed version of the BP algorithm and also its speedup analysis. We compare the performance of our algorithm with a recent work involving the vertical partitioning approach for mapping the BP algorithm onto a distributed memory multiprocessor. Our results on SUN 3/50 NOW's show that we are able to achieve better speedups by using only two communication sets and also by avoiding some redundancy in the weights computation for one training cycle of the algorithm.

[1]  Mark A. Fanty,et al.  Computing with structured connectionist networks , 1988, CACM.

[2]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[3]  Terrence J. Sejnowski,et al.  NETtalk: a parallel network that learns to read aloud , 1988 .

[4]  Minesh B. Amin,et al.  A Scalable Parallel Formulation of the Backpropagation Algorithm for Hypercubes and Related Architectures , 1994, IEEE Trans. Parallel Distributed Syst..

[5]  Soheil Shams,et al.  Efficient implementation of neural networks on the DREAM machine , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol. IV. Conference D: Architectures for Vision and Pattern Recognition,.

[6]  Rekha Govil,et al.  Neural Networks in Signal Processing , 2000 .

[7]  Tony R. Martinez,et al.  Digital Neural Networks , 1988, Proceedings of the 1988 IEEE International Conference on Systems, Man, and Cybernetics.

[8]  Hyunsoo Yoon,et al.  Multilayer Neural Networks on Distributed-Memory Multiprocessors , 1990 .

[9]  Bernd-Peter Paris,et al.  Neural networks for multiuser detection in code-division multiple-access communications , 1992, IEEE Trans. Commun..

[10]  V. K. Prasanna Kumar,et al.  Neural network simulation on a reduced-mesh-of-trees organization , 1990, Other Conferences.

[11]  B. K. Jenkins,et al.  Image restoration using a neural network , 1988, IEEE Trans. Acoust. Speech Signal Process..

[12]  Paul F. M. J. Verschure,et al.  A note on chaotic behavior in simple neural networks , 1990, Neural Networks.

[13]  Derrick H. Nguyen,et al.  Neural networks for self-learning control systems , 1990 .

[14]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[15]  Soheil Shams,et al.  Implementation of Multilayer Neural Networks on Parallel Programmable Digital Computers , 1991 .

[16]  Howard B. Demuth,et al.  Modeling neural networks on the MPP , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[17]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[18]  Lawrence A. Crowl.  How to measure, present, and compare parallel performance , 1994, IEEE Parallel & Distributed Technology: Systems & Applications.

[19]  M. Nikoonahad,et al.  Medical ultrasound imaging using neural networks , 1990 .

[20]  Guy E. Blelloch,et al.  Network Learning on the Connection Machine , 1987, IJCAI.

[21]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.

[22]  Bernd-Peter Paris,et al.  Neural Net Receivers in Multiple Access-Communications , 1988, NIPS.

[23]  Bernard Widrow,et al.  Neural nets for adaptive filtering and adaptive pattern recognition , 1988, Computer.

[24]  Etienne Deprit.  Implementing recurrent back-propagation on the connection machine , 1989, Neural Networks.

[25]  Jill P. Mesirov,et al.  The backpropagation algorithm on grid and hypercube architectures , 1990, Parallel Comput..

[26]  Seungryoul Maeng,et al.  Parallel simulation of multilayered neural networks on distributed-memory multiprocessors , 1990 .

[27]  Terrence J. Sejnowski,et al.  Learned classification of sonar targets using a massively parallel network , 1988, IEEE Trans. Acoust. Speech Signal Process..

[28]  Nam Ling,et al.  Systolic architectures for artificial neural nets , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[29]  Vaidy S. Sunderam,et al.  PVM: A Framework for Parallel Distributed Computing , 1990, Concurr. Pract. Exp..

[30]  B. Widrow,et al.  Neural networks for self-learning control systems , 1990, IEEE Control Systems Magazine.

[31]  Matthew Zeidenberg,et al.  Neural networks in artificial intelligence , 1990, Ellis Horwood series in artificial intelligence.

[32]  Simon Haykin,et al.  Classification of radar clutter using neural networks , 1991, IEEE Trans. Neural Networks.

[33]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[34]  Fa-Long Luo,et al.  Applied neural networks for signal processing , 1997 .

[35]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[36]  Geoffrey E. Hinton,et al.  Connectionist Architectures for Artificial Intelligence , 1990, Computer.

[37]  George Karypis,et al.  Introduction to Parallel Computing , 1994 .

[38]  Stephen P. Luttrell.  Image compression using a multilayer neural network , 1989, Pattern Recognit. Lett..

[39]  M. W. Roth.  Neural networks for extraction of weak targets in high clutter environments , 1989, International 1989 Joint Conference on Neural Networks.

[40]  Kunihiko Fukushima,et al.  Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.

[41]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[42]  D. Roweth,et al.  Implementing Neural Network Models on Parallel Computers , 1987, Comput. J..