Parallel implementation of back-propagation algorithm in networks of workstations

This work presents an efficient mapping scheme for the multilayer perceptron (MLP) network trained using back-propagation (BP) algorithm on network of workstations (NOWs). Hybrid partitioning (HP) scheme is used to partition the network and each partition is mapped on to processors in NOWs. We derive the processing time and memory space required to implement the parallel BP algorithm in NOWs. The performance parameters like speed-up and space reduction factor are evaluated for the HP scheme and it is compared with earlier work involving vertical partitioning (VP) scheme for mapping the MLP on NOWs. The performance of the HP scheme is evaluated by solving optical character recognition (OCR) problem in a network of ALPHA machines. The analytical and experimental performance shows that the proposed parallel algorithm has better speed-up, less communication time, and better space reduction factor than the earlier algorithm. This work also presents a simple and efficient static mapping scheme on heterogeneous system. Using divisible load scheduling theory, a closed-form expression for number of neurons assigned to each processor in the NOW is obtained. Analytical and experimental results for static mapping problem on NOWs are also presented.

[1]  C. Siva Ram Murthy,et al.  Efficient mapping of backpropagation algorithm onto a network of workstations , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[2]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[3]  Debasish Ghose,et al.  Scheduling Divisible Loads in Parallel and Distributed Systems , 1996 .

[4]  Jenq-Neng Hwang,et al.  A Unified Systolic Architecture for Artificial Neural Networks , 1989, J. Parallel Distributed Comput..

[5]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[6]  D. S. Touretzky,et al.  Neural network simulation at Warp speed: how we got 17 million connections per second , 1988, IEEE 1988 International Conference on Neural Networks.

[7]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[8]  E T. Leighton,et al.  Introduction to parallel algorithms and architectures , 1991 .

[9]  O. Egecioglu,et al.  Communication Parameter Tests and Parallel Back Propagation Algorithms on iPSC/2 Hypercube Multiprocessor , 1990, Proceedings of the Fifth Distributed Memory Computing Conference, 1990..

[10]  J.-J. Lee,et al.  A Nearly Optimal Back-Propagation Learning Algorithm on a Bus-Based Architecture , 1998, Parallel Process. Lett..

[11]  Minesh B. Amin,et al.  A Scalable Parallel Formulation of the Backpropagation Algorithm for Hypercubes and Related Architectures , 1994, IEEE Trans. Parallel Distributed Syst..

[12]  Naoyuki Fukuda,et al.  Massively parallel architectures for large scale neural network simulations , 1992, IEEE Trans. Neural Networks.

[13]  S. Jones,et al.  A Performance Model for Multilayer Neural Networks in Linear Arrays , 1994, IEEE Trans. Parallel Distributed Syst..

[14]  Sudipta Mahapatra,et al.  Mapping of Neural Network Models onto Systolic Arrays , 2000, J. Parallel Distributed Comput..

[15]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[16]  M. Nikoonahad,et al.  Medical ultrasound imaging using neural networks , 1990 .

[17]  Yi Lu Murphey,et al.  Feature extraction for a multiple pattern classification neural network system , 2002, Object recognition supported by user interaction for service robots.

[18]  Seungryoul Maeng,et al.  Parallel simulation of multilayered neural networks on distributed-memory multiprocessors , 1990 .

[19]  Mark A. Fanty,et al.  Computing with structured connectionist networks , 1988, CACM.

[20]  Sun-Yuan Kung,et al.  Digital neural networks , 1993, Prentice Hall Information and System Sciences Series.

[21]  Vaidy S. Sunderam,et al.  PVM: A Framework for Parallel Distributed Computing , 1990, Concurr. Pract. Exp..

[22]  Daniel A. Reed,et al.  Learning to Classify Parallel Input/Output Access Patterns , 2002, IEEE Trans. Parallel Distributed Syst..

[23]  Kai Hwang,et al.  Mapping Neural Networks onto Message-Passing Multicomputers , 1989, J. Parallel Distributed Comput..

[24]  Howard B. Demuth,et al.  Modeling neural networks on the MPP , 1988, Proceedings., 2nd Symposium on the Frontiers of Massively Parallel Computation.

[25]  Benjamin W. Wah,et al.  Optimal Mapping of Neural-Network Learning on Message-Passing Multicomputers , 1992, J. Parallel Distributed Comput..

[26]  V. Rego,et al.  Optimistic parallel simulation over a network of workstations , 1999, WSC'99. 1999 Winter Simulation Conference Proceedings. 'Simulation - A Bridge to the Future' (Cat. No.99CH37038).

[27]  Paramasivan Saratchandran,et al.  Parallel architectures for artificial neural networks - paradigms and implementations , 1998 .

[28]  F. Leighton,et al.  Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes , 1991 .

[29]  Ahmed El-Amawy,et al.  Algorithmic Mapping of Feedforward Neural Networks onto Multiple Bus Systems , 1997, IEEE Trans. Parallel Distributed Syst..

[30]  Viktor K. Prasanna,et al.  Algorithmic Mapping of Neural Network Models onto Parallel SIMD Machines , 1991, IEEE Trans. Computers.

[31]  Etienne Deprit.  Implementing recurrent back-propagation on the connection machine , 1989, Neural Networks.