Optimal Mapping of Neural-Network Learning on Message-Passing Multicomputers

Abstract In this paper, we study the optimal mapping of the learning process in multilayer feed-forward artificial neural networks (ANNs) on message-passing multicomputers, with an objective of minimizing the completion time of the given learning algorithm. This optimization problem is NP-hard in general and cannot be solved directly even for a small number of neurons. By observing the dominance of the computation time of a parallel neural-network learning algorithm over its communication time, we present a novel approximation algorithm for mapping large neural networks on multicomputers, given a user-specified error degree that can be tolerated in the final mapping. The target ANNs we study are learned by a static learning rule, such as the back-error-propagation learning algorithm. We study both static and dynamic mapping schemes for systems with static and dynamic background workload. Experimental results for mapping on systems with static background workload, which include a network of Sun workstations and an Intel iPSC/2 Hypercube multicomputer, are found to be very close to those predicted by analysis. Experimental results for mapping on multicomputers with dynamic background workload are obtained by simulations.

[1]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[2]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[3]  D. Roweth,et al.  Implementing Neural Network Models on Parallel Computers , 1987, Comput. J..

[4]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[5]  P. Close.  The iPSC/2 node architecture , 1988, C3P.

[6]  Lawrence D. Jackel,et al.  VLSI implementation of a neural network model , 1988, Computer.

[7]  Viktor K. Prasanna,et al.  Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines , 1991, IEEE Trans. Computers.

[8]  Mark A. Fanty,et al.  Computing with structured connectionist networks , 1988, CACM.

[9]  S. Y. King.  Parallel architectures for artificial neural nets , 1988, [1988] Proceedings. International Conference on Systolic Arrays.

[10]  Kai Hwang,et al.  Critical issues in mapping neural networks on message-passing multicomputers , 1988, ISCA '88.

[11]  Benjamin W. Wah,et al.  Optimal Mapping of Neural Networks on Multicomputers , 1990, ICPP.

[12]  S. Y. Kung,et al.  Parallel architectures for artificial neural nets , 1988, IEEE 1988 International Conference on Neural Networks.

[13]  S. F. Nugent,et al.  The iPSC/2 direct-connect communications technology , 1988, C3P.

[14]  R. Arlauskas,et al.  iPSC/2 system: a second generation hypercube , 1988, C3P.

[15]  Jeffrey D. Ullman,et al.  NP-Complete Scheduling Problems , 1975, J. Comput. Syst. Sci..