MODELS AND TRENDS IN PARALLEL PROGRAMMING

This paper introduces and discusses programming models for parallel processing and recent trends in the area of parallel programming. The paper discusses different parallel programming languages and tools that reflect various parallel computation models. These language differ in expressiveness, portability and performance. Software design and implementation largely varies by using different languages that make the programmer task easy or complex. We describe here the design goals and the main issues of parallel programming models and languages belonging to the following categories: shared-space based languages, message-based languages, parallel toolkits, data-parallel languages, parallel declarative languages, parallel object-oriented languages, and parallel composition-based languages. Tools and languages such as HPF, Linda, Java, OpenMP, PVM, MPI, Parallel C+ +, Sisal, Orca, Mentat, SkieCL, BSP and others are described in some detail. Their main features for design and implementation of high performance applications are discussed. Finally, we outline directions of research and development in the parallel programming area with a special attention to novel approaches based on high-level programming structures that make transparent to the users the architectural details of parallel computing machines.

[1]  K. Mani Chandy,et al.  CC++: A Declarative Concurrent Object Oriented Programming Notation , 1993 .

[2]  Jayadev Misra,et al.  An Object Model for Multiprogramming , 1998, IPPS/SPDP Workshops.

[3]  Bradley C. Kuszmaul,et al.  Cilk: an efficient multithreaded runtime system , 1995, PPOPP '95.

[4]  K BennettJohn,et al.  Implementation and performance of Munin , 1991 .

[5]  Nicholas Carriero,et al.  Applications experience with Linda , 1988, PPoPP 1988.

[6]  Gul A. Agha,et al.  Actors: A unifying model for parallel and distributed computing , 1999, J. Syst. Archit..

[7]  An Hong.  Parallel Programming Models and Languages , 2002 .

[8]  Alan L. Cox,et al.  TreadMarks: shared memory computing on networks of workstations , 1996 .

[9]  Andrew S. Grimshaw,et al.  Easy-to-use object-oriented parallel processing with Mentat , 1993, Computer.

[10]  Domenico Talia,et al.  Programming cellular automata algorithms on parallel computers , 1999, Future Gener. Comput. Syst..

[11]  Adam Ferrari,et al.  JPVM: network parallel computing in Java , 1997, Concurr. Pract. Exp..

[12]  Steve Gregory,et al.  Experiments with Speculative Parallelism in Parlog , 1993, ILPS.

[13]  Piyush Mehrotra,et al.  Language support for multidisciplinary applications , 1998 .

[14]  Andrew S. Grimshaw,et al.  Wide-Area Computing: Resource Sharing on a Large Scale , 1999, Computer.

[15]  Domenico Talia.  Parallel computation still not ready for the mainstream , 1997, CACM.

[16]  Jack Dongarra,et al.  PVM and HeNCE: tools for heterogeneous network computing , 1993 .

[17]  Gruia-Catalin Roman,et al.  A shared dataspace language supporting large-scale concurrency , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[18]  Philip J. Hatcher,et al.  Data-parallel programming on multicomputers , 1990, IEEE Software.

[19]  Pascal Fradet,et al.  Structured Gamma , 1998, Sci. Comput. Program..

[20]  James C. Browne,et al.  Support for Implementation of Evolutionary Concurrent Systems in Concurrent Programming Languages , 1996, Euro-Par, Vol. I.

[21]  K. Mani Chandy,et al.  The Derivation of Compositional Programs , 1992, JICSLP.

[22]  Murray Cole,et al.  Algorithmic Skeletons: Structured Management of Parallel Computation , 1989 .

[23]  Bradford L. Chamberlain,et al.  The case for high-level parallel programming in ZPL , 1998 .

[24]  Dennis Gannon,et al.  A capabilities based communication model for high-performance distributed applications: The Open HPC++ approach , 1999, Proceedings 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing. IPPS/SPDP 1999.

[25]  Barbara Liskov,et al.  Distributed programming in Argus , 1988, CACM.

[26]  Guy E. Blelloch,et al.  Programming parallel algorithms , 1996, CACM.

[27]  Yike Guo,et al.  Functional Skeletons for Parallel Coordination , 1995, Euro-Par.

[28]  Domenico Talia,et al.  The CARPET Programming Environment for Solving Scientific Problems on Parallel Computers , 1998, Scalable Comput. Pract. Exp..

[29]  Bradford Nichols,et al.  Pthreads programming - a POSIX standard for better multiprocessing , 1996 .

[30]  Per Brinch Hansen,et al.  Java's insecure parallelism , 1999, SIGP.

[31]  Anthony Skjellum,et al.  The JMPI Commercial Message Passing Environment and Specification: Requirements, Design, Motivatio , 1998 .

[32]  K. Mani Chandy,et al.  Fortran M: A Language for Modular Parallel Programming , 1995, J. Parallel Distributed Comput..

[33]  Laxmikant V. Kalé,et al.  CHARM++: a portable concurrent object oriented system based on C++ , 1993, OOPSLA '93.

[34]  Doug Lea,et al.  Concurrent programming in Java - design principles and patterns , 1996, Java series.

[35]  John H. Reppy,et al.  CML: A Higher-Order Concurrent Language , 1991, PLDI.

[36]  Jaime Seguel,et al.  A Functional Approach to Radix-R FFTs , 2001, Scalable Comput. Pract. Exp..

[37]  Akinori Yonezawa,et al.  Towards Foundations of Concurrent Object-Oriented Programming-Types and Language Design , 1995, Theory Pract. Object Syst..

[38]  Paul G. Spirakis,et al.  BSP vs LogP , 1996, SPAA '96.

[39]  Ian T. Foster,et al.  Productive Parallel Programming: The PCN Approach , 1995, Sci. Program..

[40]  Robert H. Halstead,et al.  Parallel Symbolic Computing , 1986, Computer.

[41]  Paddy Nixon,et al.  Bridging Boundaries: CORBA in Perspective , 1997, IEEE Internet Comput..

[42]  David B. Skillicorn,et al.  Models and languages for parallel computation , 1998, CSUR.

[43]  Geoffrey C. Fox,et al.  Towards a Java Environment for SPMD Programming , 1998, Euro-Par.

[44]  Henri E. Bal,et al.  Orca: A Language For Parallel Programming of Distributed Systems , 1992, IEEE Trans. Software Eng..

[45]  Narain H. Gehani,et al.  Implementing concurrent C , 1992, Softw. Pract. Exp..

[46]  Adam J. Ferrari.  JPVM: Network Parallel Computing in Java , 1997 .

[47]  Jack J. Dongarra,et al.  Recent Enhancements To Pvm , 1995, Int. J. High Perform. Comput. Appl..

[48]  Henri E. Bal,et al.  A task- and data-parallel programming language based on shared objects , 1998, TOPL.

[49]  John H. Reppy,et al.  CML: A higher concurrent language , 1991, PLDI '91.

[50]  Willy Zwaenepoel,et al.  Munin: distributed shared memory based on type-specific memory coherence , 1990, PPOPP '90.

[51]  Marco Vanneschi,et al.  PQE2000: HPC tools for industrial applications , 1998, IEEE Concurr..

[52]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[53]  Leslie Lamport,et al.  How to Make a Multiprocessor Computer That Correctly Executes Multiprocess Programs , 2016, IEEE Transactions on Computers.

[54]  Klaus-Peter Löhr,et al.  Object-Oriented Concurrent Programming , 1992, TOOLS.

[55]  Ian T. Foster,et al.  The Nexus Approach to Integrating Multithreading and Communication , 1996, J. Parallel Distributed Comput..

[56]  Tomás Lang,et al.  Architectural support for goal management in flat concurrent Prolog , 1992, Computer.

[57]  David B. Skillicorn,et al.  Questions and Answers about BSP , 1997, Sci. Program..

[58]  Michael Metcalf,et al.  High performance Fortran , 1995 .

[59]  David B. Skillicorn,et al.  Programming languages for parallel processing , 1995 .

[60]  Ken Wakita,et al.  D'Caml: Native Support for Distributed ML Programming in Heterogeneous Environment , 1999, Euro-Par.

[61]  Willy Zwaenepoel,et al.  Implementation and performance of Munin , 1991, SOSP '91.

[62]  Jacques Chassin de Kergommeaux,et al.  Parallel logic programming systems , 1994, CSUR.

[63]  Torsten Suel,et al.  BSPlib: The BSP programming library , 1998, Parallel Comput..

[64]  Domenico Talia,et al.  A Survey of Parlog and Concurrent Prolog: The Integration of Logic and Parallelism , 1993, Comput. Lang..

[65]  Steve Gregory,et al.  A Declarative Approach to Concurrent Programming , 1997, PLILP.

[66]  Philip J. Hatcher,et al.  Data-Parallel Programming on MIMD Computers , 1991, IEEE Trans. Parallel Distributed Syst..

[67]  Louise E. Moser,et al.  Analysis of timing hazards in Ada programs , 1993, CSC '93.