A user-centric perspective on parallel programming with focus on openMP

The process of developing software that takes advantage of multiple processors is commonly referred to as parallel programming. For various reasons, this process is much harder than the sequential case. For decades, parallel programming has been a problem for a small niche only: engineers working on parallelizing mostly numerical applications in High Performance Computing. This has changed with the advent of multi-core processors in mainstream computer architectures. Parallel programming in our days becomes a problem for a much larger group of developers. The main objective of this thesis was to find ways to make parallel programming easier for them. Different aims were identified in order to reach the objective: research the state of the art of parallel programming today, improve the education of software developers about the topic, and provide programmers with powerful abstractions to make their work easier. To reach these aims, several key steps were taken. To start with, a survey was conducted among parallel programmers to find out about the state of the art. More than 250 people participated, yielding results about the parallel programming systems and languages in use, as well as about common problems with these systems. Furthermore, a study was conducted in university classes on parallel programming. It resulted in a list of frequently made mistakes that were analyzed and used to create a programmers' checklist to avoid them in the future. For programmers' education, an online resource was setup to collect experiences and knowledge in the field of parallel programming – called the Parawiki. Another key step in this direction was the creation of the Thinking Parallel weblog, where more than 50.000 readers to date have read essays on the topic. For the third aim (powerful abstractions), it was decided to concentrate on one parallel programming system: OpenMP. Its ease of use and high level of abstraction were the most important reasons for this decision. Two different research directions were pursued. The first one resulted in a parallel library called AthenaMP. It contains so-called generic components, derived from design patterns for parallel programming. These include functionality to enhance the locks provided by OpenMP, to perform operations on large amounts of data (data-parallel programming), and to enable the implementation of irregular algorithms using task pools. AthenaMP itself serves a triple role: the components are well-documented and can be used directly in programs, it enables developers to study the source code and learn from it, …

[1]  Barbara Chapman,et al.  Using OpenMP - portable shared memory parallel programming , 2007, Scientific and engineering computation.

[2]  Douglas C. Schmidt,et al.  Pattern-Oriented Software Architecture, Patterns for Concurrent and Networked Objects , 2013 .

[3]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[4]  Michael Wolfe,et al.  High performance compilers for parallel computing , 1995 .

[5]  Timothy G. Mattson,et al.  How good is OpenMP , 2003, Sci. Program..

[6]  Claudia Fohry,et al.  Common Mistakes in OpenMP and How to Avoid Them - A Collection of Best Practices , 2005, IWOMP.

[7]  Claudia Fohry,et al.  A Comparison of Task Pool Variants in OpenMP and a Proposal for a Solution to the Busy Waiting Problem , 2005, IWOMP.

[8]  Scott Meyers,et al.  C++ and the Perils of Double-Checked Locking ∗ , 2004 .

[9]  Jack Dongarra,et al.  PVM: Parallel virtual machine: a users' guide and tutorial for networked parallel computing , 1995 .

[10]  John H. Baldwin.  Locking in the Multithreaded FreeBSD Kernel , 2002, BSDCon.

[11]  Willy Zwaenepoel,et al.  OpenMP on Networks of Workstations , 1998, Proceedings of the IEEE/ACM SC98 Conference.

[12]  Thomas Rauber,et al.  A comparison of task pools for dynamic load balancing of irregular algorithms , 2004, Concurr. Comput. Pract. Exp..

[13]  David Holmes,et al.  Java Concurrency in Practice , 2006 .

[14]  Joel Spolsky,et al.  Joel on Software , 2004, Apress.

[15]  Nancy M. Amato,et al.  STAPL: An Adaptive, Generic Parallel C++ Library , 2001, LCPC.

[16]  Dennis Gannon,et al.  HPC++: experiments with the parallel standard template library , 1997, ICS '97.

[17]  Claudia Leopold,et al.  Parallel and Distributed Computing: A Survey of Models, Paradigms and Approaches , 2008 .

[18]  C. Leopold,et al.  A User ’ s Experience with Parallel Sorting and OpenMP , 2004 .

[19]  Gagan Agrawal,et al.  Porting and performance evaluation of irregular codes using OpenMP , 2000, Concurr. Pract. Exp..

[20]  Daniel C. Howe,et al.  Free on-line dictionary of computing , 2006 .

[21]  Claudia Fohry,et al.  Implementing Irregular Parallel Algorithms with OpenMP , 2006, Euro-Par.

[22]  Paul Petersen,et al.  Flexible control structures for parallelism in OpenMP , 2000, Concurr. Pract. Exp..

[23]  Claudia Fohry,et al.  Parallelizing a Real-Time Steering Simulation for Computer Games with OpenMP , 2007, PARCO.

[24]  J. Mark Bull,et al.  A microbenchmark suite for OpenMP 2.0 , 2001, CARN.

[25]  C. Leopold,et al.  Programming for Malleability with Hybrid MPI-2 and OpenMP – Experiences with a Simulation Program for Global Water Prognosis , 2006 .

[26]  E. Ayguade,et al.  Scaling Irregular Parallel Codes with Minimal Programming Effort , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[27]  C. Q. Lee,et al.  The Computer Journal , 1958, Nature.

[28]  Claudia Leopold,et al.  Evaluating the state of the art of parallel programming systems , 2005 .

[29]  Claudia Leopold,et al.  Observations on the Publicity and Usage of Parallel Programming Systems and Languages: A Survey Approach , 2007 .

[30]  Angela C. Sodan.  Message-passing and shared-data programming models - wish vs. reality , 2005, 19th International Symposium on High Performance Computing Systems and Applications (HPCS'05).

[31]  Claudia Fohry,et al.  Problems, Workarounds and Possible Solutions Implementing the Singleton Pattern with C++ and OpenMP , 2007, IWOMP.

[32]  Timothy G. Mattson,et al.  Patterns for parallel programming , 2004 .

[33]  J. Pasciak,et al.  Computer solution of large sparse positive definite systems , 1982 .

[34]  Michael J. Quinn,et al.  Parallel programming in C with MPI and OpenMP , 2003 .

[35]  Rohit Chandra,et al.  Parallel programming in openMP , 2000 .

[36]  Eduard Ayguadé,et al.  Nanos mercurium: A research compiler for OpenMP , 2004 .

[37]  Plateau de Moulon,et al.  Comparison of OpenMP and Classical Multi-Threading Parallelization for Regular and Irregular Algorithms , 2000 .

[38]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[39]  Joe Armstrong,et al.  Programming Erlang: Software for a Concurrent World , 1993 .

[40]  Claudia Fohry,et al.  Generic Locking and Deadlock-Prevention with C++ , 2007, PARCO.

[41]  M. F.,et al.  Bibliography , 1985, Experimental Gerontology.

[42]  Bronis R. de Supinski,et al.  The OpenMP Memory Model , 2005, IWOMP.

[43]  David R. Butenhof.  Programming with POSIX threads , 1993 .

[44]  Andrei Alexandrescu,et al.  Modern C++ design: generic programming and design patterns applied , 2001 .

[45]  Claudia Fohry,et al.  Implementing Data-Parallel Patterns for Shared Memory with OpenMP , 2007, PARCO.

[46]  Christian Terboven,et al.  Automatic Scoping of Variables in Parallel Regions of an OpenMP Program , 2004, WOMPAT.

[47]  Ian Foster,et al.  Designing and building parallel programs , 1994 .

[48]  Scott Meyers,et al.  Effective C++: 55 Specific Ways to Improve Your Programs and Designs (3rd Edition) , 1991 .

[49]  Alejandro Duran,et al.  A Proposal for Task Parallelism in OpenMP , 2007, IWOMP.

[50]  Claudia Fohry,et al.  Observations on MPI-2 Support for Hybrid Master/Slave Applications in Dynamic and Heterogeneous Environments , 2006, PVM/MPI.