Benchmarking Usability and Performance of Multicore Languages

Developers face a wide choice of programming languages and libraries supporting multicore computing. Ever more diverse paradigms for expressing parallelism and synchronization become available while their influence on usability and performance remains largely unclear. This paper describes an experiment comparing four markedly different approaches to parallel programming: Chapel, Cilk, Go, and Threading Building Blocks (TBB). Each language is used to implement sequential and parallel versions of six benchmark programs. The implementations are then reviewed by notable experts in the language, thereby obtaining reference versions for each language and benchmark. The resulting pool of 96 implementations is used to compare the languages with respect to source code size, coding time, execution time, and speedup. The experiment uncovers strengths and weaknesses in all approaches, facilitating an informed selection of a language under a particular set of requirements. The expert review step furthermore highlights the importance of expert knowledge when using modern parallel programming approaches.

[1]  Jonathan Schaeffer,et al.  An experiment to measure the usability of parallel programming systems , 1996, Concurr. Pract. Exp..

[2]  Jeffrey H. Meyerson,et al.  The Go Programming Language , 2014, IEEE Softw..

[3]  John T. Feo.  Comparative Study of Parallel Programming Languages: The Salishan Problems , 1992 .

[4]  Danny Dig,et al.  How do developers use parallel libraries? , 2012, SIGSOFT FSE.

[5]  Bradley C. Kuszmaul,et al.  Cilk: an efficient multithreaded runtime system , 1995, PPOPP '95.

[6]  Bradford L. Chamberlain,et al.  Parallel Programmability and the Chapel Language , 2007, Int. J. High Perform. Comput. Appl..

[7]  Victor Pankratius,et al.  Combining functional and imperative programming for multicore software: An empirical study evaluating Scala and Java , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[8]  C. A. R. Hoare,et al.  Communicating sequential processes , 1978, CACM.

[9]  David B. Skillicorn,et al.  Models and languages for parallel computation , 1998, CSUR.

[10]  James Demmel,et al.  the Parallel Computing Landscape , 2022 .

[11]  Gregory V. Wilson.  Assessing the Usability of Parallel Programming Systems: The Cowichan Problems , 1994 .

[12]  Walter F. Tichy,et al.  Measuring High Performance Computing Productivity , 2004, Int. J. High Perform. Comput. Appl..

[13]  Uzi Vishkin,et al.  A pilot study to compare programming effort for two parallel programming models , 2007, J. Syst. Softw..

[14]  Ken Kennedy,et al.  Defining and Measuring the Productivity of Programming Languages , 2004, Int. J. High Perform. Comput. Appl..

[15]  Sebastian Nanz,et al.  Examining the Expert Gap in Parallel Programming , 2013, Euro-Par.

[16]  Vivek Sarkar,et al.  An Experiment in Measuring the Productivity of Three Parallel Programming Languages , 2007 .

[17]  James Reinders,et al.  Intel threading building blocks - outfitting C++ for multi-core processor parallelism , 2007 .

[18]  Bertrand Meyer,et al.  Design of an empirical study for comparing the usability of concurrent programming languages , 2013, Inf. Softw. Technol..

[19]  Emmett Witchel,et al.  Is transactional programming actually easier? , 2010, PPoPP '10.

[20]  Jeffrey C. Carver,et al.  Parallel Programmer Productivity: A Case Study of Novice Parallel Programmers , 2005, ACM/IEEE SC 2005 Conference (SC'05).

[21]  Murray Cole,et al.  Algorithmic Skeletons: Structured Management of Parallel Computation , 1989 .

[22]  George Almási.  PGAS (Partitioned Global Address Space) Languages , 2011, Encyclopedia of Parallel Computing.

[23]  Partitioned Global Address Space (PGAS) Languages , 2011, Encyclopedia of Parallel Computing.

[24]  Mohamed M. Zahran,et al.  Productivity analysis of the UPC language , 2004, 18th International Parallel and Distributed Processing Symposium, 2004. Proceedings..

[25]  Henri E. Bal,et al.  A comparative study of five parallel programming languages , 1992, Future Gener. Comput. Syst..