Statistical Analysis of Computational Tests of Algorithms and Heuristics

Statistical analysis is a powerful tool to apply when evaluating the performance of computer implementations of algorithms and heuristics. Yet many computational studies in the literature do not use this tool to maximum effectiveness. This paper examines the types of data that arise in computational comparisons and presents appropriate techniques for analyzing such data sets. Case studies of computational tests from the open literature are re-evaluated using the proposed methods in order to illustrate the value of statistical analysis for gaining insight into the behavior of the tested algorithms.

[1]  Catherine C. McGeoch.  Feature Article - Toward an Experimental Method for Algorithm Simulation , 1996, INFORMS J. Comput..

[2]  J. M. Mulvey,et al.  A Critical Review of Comparisons of Mathematical Programming Algorithms and Software (1953-1977). , 1978, Journal of research of the National Bureau of Standards.

[3]  I. Lustig,et al.  Computational experience with a primal-dual interior point method for linear programming , 1991 .

[4]  M. E. Shayan.  A methodology for algorithm comparison in mathematical programming , 1987 .

[5]  Andreas T. Ernst,et al.  Efficient algorithms for the uncapac-itated single allocation p-hub median problem , 1996 .

[6]  P. B. Saunders,et al.  Methodology and analysis for comparing discrete linear l1 approximation codes , 1977 .

[7]  Richard J. Anderson,et al.  The role of experiment in the theory of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[8]  Klaus Schittkowski,et al.  Test examples for nonlinear programming codes , 1980 .

[9]  Philip H. Ramsey.  Nonparametric Statistical Methods , 1974, Technometrics.

[10]  Martin W. P. Savelsbergh,et al.  Lifted Cover Inequalities for 0-1 Integer Programs: Complexity , 1999, INFORMS J. Comput..

[11]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[12]  R. Weiner.  Lecture Notes in Economics and Mathematical Systems , 1985 .

[13]  Egon Balas,et al.  An Algorithm for the Three-Index Assignment Problem , 1991, Oper. Res..

[14]  Stephen G. Nash,et al.  Guidelines for reporting results of computational experiments. Report of the ad hoc committee , 1991, Math. Program..

[15]  S. Hora.  Statistical Inference Based on Ranks , 1986 .

[16]  Douglas R. Shier.  Commentary - On Algorithm Analysis , 1996, INFORMS J. Comput..

[17]  William W White.  Computers and mathematical programming , 1978 .

[18]  Wayne Nelson,et al.  Applied life data analysis , 1983 .

[19]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[20]  Ravindra K. Ahuja,et al.  Use of Representative Operation Counts in Computational Testing of Algorithms , 1996, INFORMS J. Comput..

[21]  A. Miele,et al.  ON THE COMPARATIVE EVALUATION OF ALGORITHMS FOR MATHEMATICAL PROGRAMMING PROBLEMS , 1982 .

[22]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[23]  C. Borror.  Nonparametric Statistical Methods, 2nd, Ed. , 2001 .

[24]  Klaus Schittkowski,et al.  More test examples for nonlinear programming codes , 1981 .

[25]  Mohammad M. Amini,et al.  A rigorous computational comparison of alternative solution methods for the generalized assignment problem , 1994 .

[26]  Pierre Hansen,et al.  Constrained Nonlinear 0-1 Programming , 1989 .

[27]  Andreas T. Ernst,et al.  An Exact Solution Approach Based on Shortest-Paths for p-Hub Median Problems , 1998, INFORMS J. Comput..

[28]  Mauricio G. C. Resende,et al.  Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[29]  Yves Rochat,et al.  Probabilistic diversification and intensification in local search for vehicle routing , 1995, J. Heuristics.

[30]  Susan A. Murphy,et al.  The Theil-Sen Estimator with Doubly Censored Data and Applications to Astronomy , 1995 .

[31]  Leonie Kohl,et al.  Fundamental Concepts in the Design of Experiments , 2000 .

[32]  C. Ireland.  Fundamental concepts in the design of experiments , 1964 .

[33]  Pierre Hansen,et al.  State-of-the-Art Survey - Constrained Nonlinear 0-1 Programming , 1993, INFORMS J. Comput..

[34]  Peter J. Rousseeuw,et al.  Robust regression and outlier detection , 1987 .

[35]  Arno Jewett,et al.  A Critical Review of , 1964 .

[36]  Harvey J. Greenberg.  Computational Testing: Why, How and How Much , 1990, INFORMS J. Comput..

[37]  D. Cox,et al.  An Analysis of Transformations , 1964 .

[38]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.

[39]  Robert J. Vanderbei,et al.  Linear Programming: Foundations and Extensions , 1998, Kluwer international series in operations research and management service.

[40]  Catherine C. McGeoch.  Rejoinder - Challenges in Algorithm Simulation , 1996, INFORMS J. Comput..

[41]  Sidney Addelman,et al.  trans-Dimethanolbis(1,1,1-trifluoro-5,5-dimethylhexane-2,4-dionato)zinc(II) , 2008, Acta crystallographica. Section E, Structure reports online.

[42]  R. H. F. Jackson,et al.  In Pursuit of a Methodology for Testing Mathematical Programming Software , 1982 .

[43]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[44]  James B. Orlin.  Commentary - On Experimental Methods for Algorithm Simulation , 1996, INFORMS J. Comput..

[45]  David C. Hoaglin,et al.  Exploratory Data Analysis in a Study of the Performance of Nonlinear Optimization Routines , 1982, TOMS.

[46]  John M. Mulvey,et al.  Reporting computational experiments in mathematical programming , 1978, Math. Program..

[47]  Martin W. P. Savelsbergh,et al.  Lifted Cover Inequalities for 0-1 Integer Programs: Computation , 1998, INFORMS J. Comput..

[48]  Pierre L'Ecuyer.  Commentary - Simulation of Algorithms for Performance Analysis , 1996, INFORMS J. Comput..

[49]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[50]  Bruce L. Golden,et al.  Experimentation in optimization , 1986 .

[51]  Eitan Zemel.  Measuring the Quality of Approximate Solutions to Zero-One Programming Problems , 1981, Math. Oper. Res..

[52]  L. Nazareth,et al.  Evaluation of unconstrained optimization routines , 1977 .

[53]  Richard S. Barr,et al.  Network Reoptimization Algorithms: A Statistically Designed Comparison , 1993, INFORMS J. Comput..

[54]  V. Barnett,et al.  Applied Linear Statistical Models , 1975 .

[55]  Catherine C. McGeoch.  Analyzing algorithms by simulation: variance reduction techniques and simulation speedups , 1992, CSUR.

[56]  John M. Mulvey,et al.  Guidelines for Reporting Computational Experiments in Mathematical Programming , 1978 .

[57]  H. J. Arnold.  Introduction to the Practice of Statistics , 1990 .

[58]  Monica C. Jackson,et al.  Introduction to the Practice of Statistics , 2001 .