Feature Article - Reporting Computational Experiments with Parallel Algorithms: Issues, Measures, and Experts' Opinions

Accompanying the increasing availability of parallel computing technology is a corresponding growth of research into the development, implementation, and testing of parallel algorithms. This paper examines issues involved in reporting on the empirical testing of parallel mathematical programming algorithms, both optimizing and heuristic. We examine the appropriateness of various performance metrics and explore the effects of testing variability, machine influences, testing biases, and the effects of tuning parameters. Some of these difficulties were explored further in a survey sent to leading computational mathematical programming researchers for their reactions and suggestions. A summary of the survey and proposals for conscientious reporting are presented. INFORMS Journal on Computing , ISSN 1091-9856, was published as ORSA Journal on Computing from 1989 to 1995 under ISSN 0899-1499.

[1]  Charles L. Seitz,et al.  The cosmic cube , 1985, CACM.

[2]  Jeffery L. Kennington,et al.  Generalized Networks: Parallel Algorithms and an Empirical Analysis , 1992, INFORMS J. Comput..

[3]  Donald L. Miller,et al.  Solution of large dense transportation problems using a parallel primal algorithm , 1990 .

[4]  Nicholas Carriero,et al.  How to write parallel programs: a guide to the perplexed , 1989, CSUR.

[5]  P. John.  Statistical Design and Analysis of Experiments , 1971 .

[6]  Stavros A. Zenios,et al.  Proximal minimizations with D-functions and the massively parallel solution of linear network programs , 1993, Comput. Optim. Appl..

[7]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[8]  Mauricio G. C. Resende,et al.  A Greedy Randomized Adaptive Search Procedure for Maximum Independent Set , 1994, Oper. Res..

[9]  Michael J. Quinn,et al.  Designing Efficient Algorithms for Parallel Computers , 1987 .

[10]  J. M. Mulvey,et al.  A Critical Review of Comparisons of Mathematical Programming Algorithms and Software (1953-1977). , 1978, Journal of research of the National Bureau of Standards.

[11]  Sartaj Sahni,et al.  Anomalies in Parallel Branch-and-Bound Algorithms , 1984 .

[12]  Ray Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[13]  W. Daniel Hillis,et al.  The connection machine , 1985 .

[14]  Xiaoye S. Li,et al.  Data-level parallel solution of min-cost network flow problems using ϵ-relaxations , 1994 .

[15]  David J. DeWitt,et al.  Parallel database systems: the future of high performance database systems , 1992, CACM.

[16]  Panos M. Pardalos,et al.  Computational Experience with Parallel Algorithms for solving the quadratic Assignment Problem , 1992, Computer Science and Operations Research.

[17]  Stavros A. Zenios,et al.  Parallel Numerical Optimization: Current Status and an Annotated Bibliography , 1989, INFORMS J. Comput..

[18]  Chita R. Das,et al.  A parallel branch-and-bound algorithm for MIN-based multiprocessors , 1991, SIGMETRICS '91.

[19]  R. Michael Hord.  Parallel supercomputing in SIMD architectures , 1990 .

[20]  Richard S. Barr,et al.  Network Reoptimization Algorithms: A Statistically Designed Comparison , 1993, INFORMS J. Comput..

[21]  Dimitri P. Bertsekas,et al.  Parallel computing in network optimization , 1994 .

[22]  P. B. Saunders,et al.  Methodology and analysis for comparing discrete linear l1 approximation codes , 1977 .

[23]  G. R. Withers,et al.  Computing performance as a function of the speed, quantity, and cost of the processors , 1989, Proceedings of the 1989 ACM/IEEE Conference on Supercomputing (Supercomputing '89).

[24]  R. V. Helgason,et al.  Algorithms for network programming , 1980 .

[25]  John L. Gustafson,et al.  The consequences of fixed time performance measurement , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[26]  Jörg Peters,et al.  The network simplex method on a multiprocessor , 1990, Networks.

[27]  Richard S. Barr,et al.  On Reporting the Speedup of Parallel Algorithms: a Survey of Issues and Experts , 1992, Computer Science and Operations Research.

[28]  John M. Mulvey,et al.  Guidelines for Reporting Computational Experiments in Mathematical Programming , 1978 .

[29]  B. Gillett,et al.  A parallel integer linear programming algorithm , 1988 .

[30]  Robert E. Benner,et al.  Development of Parallel Methods for a $1024$-Processor Hypercube , 1988 .

[31]  Harvey J. Greenberg.  Computational Testing: Why, How and How Much , 1990, INFORMS J. Comput..

[32]  E. L. Lusk,et al.  Use of monitors in FORTRAN: a tutorial on the barrier, self-scheduling DO-loop, and askfor monitors , 1985 .

[33]  Michael J. Flynn,et al.  Very high-speed computing systems , 1966 .

[34]  J. Pekny,et al.  Results from a parallel branch and bound algorithm for the asymmetric traveling salesman problem , 1989 .