MIMD implementation of neural networks through pipelined, parallel communication trees

Hardware implementations and single-instruction-stream/multiple-data-stream (SIMD) simulations are relatively inflexible, while multiple-instruction-stream (MIMD) simulations often trade their flexibility for increased efficiency. An alternative technique is presented which is based on pipelining fewer but larger messages through parallel, broadcast/accumulate trees. This method exploits both the structural parallelism of neural networks and the data parallelism of neural algorithms. The mapping is flexible to changes in the network architecture and learning algorithm and is suited to a variety of computer configurations. Experimental results show a higher efficiency than similar implementation.<<ETX>>

[1]  Mark A. Fanty,et al.  Computing with structured connectionist networks , 1988, CACM.

[2]  Gul A. Agha,et al.  ACTORS - a model of concurrent computation in distributed systems , 1985, MIT Press series in artificial intelligence.

[3]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[4]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[5]  Hyunsoo Yoon,et al.  A distributed backpropagation algorithm of neural networks on distributed-memory multiprocessors , 1990, [1990 Proceedings] The Third Symposium on the Frontiers of Massively Parallel Computation.

[6]  K. W. Przytula,et al.  Mapping of neural networks onto programmable parallel machines , 1990, IEEE International Symposium on Circuits and Systems.

[7]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[8]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[9]  Guy E. Blelloch,et al.  Network Learning on the Connection Machine , 1987, IJCAI.

[10]  G. A. Kohring.  Large Scale Neural Network Simulations , 1990 .

[11]  Benjamin W. Wah,et al.  Optimal Mapping of Neural Networks on Multicomputers , 1990, ICPP.

[12]  Mark E. Nelson,et al.  Simulating neurons and networks on parallel computers , 1989 .