Efficient Mapping of ANNs on Hypercube Massively Parallel Machines

This paper presents a technique for mapping artificial neural networks (ANNs) on hypercube massively parallel machines. The paper starts by synthesizing a parallel structure, the mesh-of-appendixed-trees (MAT), for fast ANN implementation. Then, it presents a recursive procedure to embed the MAT structure into the hypercube topology. This procedure is used as the basis for an efficient mapping of ANN computations on hypercube systems. Both the multilayer feedforward with backpropagation (FFBP) and the Hopfield ANN models are considered. Algorithms to implement the recall and the training phases of the FFBP model as well as the recall phase of the Hopfield model are provided. The major advantage of our technique is high performance. Unlike the other techniques presented in the literature which require O(n) time, where N is the size of the largest layer, our implementation requires only O(log N) time. Moreover, it allows pipelining of more than one input pattern and thus further improves the performance. >

[1]  Ulrich Ramacher,et al.  SYNAPSE - A Neurocomputer that Synthesizes Neural Algorithms on a Parallel Systolic Engine , 1992, J. Parallel Distributed Comput..

[2]  Jill P. Mesirov,et al.  An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2 , 1989, NIPS.

[3]  Magdy A. Bayoumi,et al.  An application-specific array architecture for feedforward with backpropagation ANNs , 1993, Proceedings of International Conference on Application Specific Array Processors (ASAP '93).

[4]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[5]  Eiji Fujiwara,et al.  Single b-bit byte error correcting and double bit error detecting codes for high-speed memory systems , 1992, [1992] Digest of Papers. FTCS-22: The Twenty-Second International Symposium on Fault-Tolerant Computing.

[6]  Anargyros Krikelis,et al.  Implementing Neural Networks with the Associative String Processor , 1991 .

[7]  Viktor K. Prasanna,et al.  Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines , 1991, IEEE Trans. Computers.

[8]  M. A. Bayoumi,et al.  A reconfigurable 'ANN' architecture , 1992, [Proceedings] 1992 IEEE International Symposium on Circuits and Systems.

[9]  Chin-Long Chen,et al.  Error-Correcting Codes with Byte Error-Detection Capability , 1983, IEEE Trans. Computers.

[10]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[11]  Jenq-Neng Hwang,et al.  Parallel algorithms/architectures for neural networks , 1989, J. VLSI Signal Process..

[12]  C. Chen,et al.  Systolic array implementations of neural nets on the MasPar MP-1 massively parallel processor , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  Guy E. Blelloch,et al.  An implementation of network learning on the Connection Machine , 1988 .

[14]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[15]  S. Y. King.  Parallel architectures for artificial neural nets , 1988, [1988] Proceedings. International Conference on Systolic Arrays.

[16]  Jeff A. Bilmes,et al.  The Ring Array Processor: A Multiprocessing Peripheral for Connection Applications , 1992, J. Parallel Distributed Comput..

[17]  S. Tomboulian.  Overview and extensions of a system for routing directed graphs on SIMD architectures , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[18]  Benjamin W. Wah,et al.  Optimal Mapping of Neural Networks on Multicomputers , 1990, ICPP.

[19]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[20]  Robert H. Deng,et al.  Decoding of DBEC-TBED Reed-Solomon Codes , 1987, IEEE Transactions on Computers.

[21]  Bertil Svensson,et al.  Using and Designing Massively Parallel Computers for Artificial Neural Neural Networks , 1992, J. Parallel Distributed Comput..

[22]  J. R. Brown,et al.  Artificial neural network on a SIMD architecture , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[23]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[24]  Benjamin W. Wah,et al.  Optimal Mapping of Neural-Network Learning on Message-Passing Multicomputers , 1992, J. Parallel Distributed Comput..

[25]  K. W. Przytula,et al.  Mapping of neural networks onto programmable parallel machines , 1990, IEEE International Symposium on Circuits and Systems.

[26]  M. Duranton,et al.  Learning on VLSI: a general purpose digital neurochip , 1989, International 1989 Joint Conference on Neural Networks.

[27]  Eiji Fujiwara,et al.  Error-control coding for computer systems , 1989 .

[28]  Kemal Efe,et al.  Embedding Mesh of Trees in the Hypercube , 1991, J. Parallel Distributed Comput..

[29]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[30]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .