Experimental Evaluation of Heuristic Optimization Algorithms: A Tutorial

Heuristic optimization algorithms seek good feasible solutions to optimization problems in circumstances where the complexities of the problem or the limited time available for solution do not allow exact solution. Although worst case and probabilistic analysis of algorithms have produced insight on some classic models, most of the heuristics developed for large optimization problem must be evaluated empirically—by applying procedures to a collection of specific instances and comparing the observed solution quality and computational burden.This paper focuses on the methodological issues that must be confronted by researchers undertaking such experimental evaluations of heuristics, including experimental design, sources of test instances, measures of algorithmic performance, analysis of results, and presentation in papers and talks. The questions are difficult, and there are no clear right answers. We seek only to highlight the main issues, present alternative ways of addressing them under different circumstances, and caution about pitfalls to avoid.

[1]  Charles H. Reilly,et al.  Multivariate composite distributions for coefficients in synthetic optimization problems , 2000, Eur. J. Oper. Res..

[2]  B. Golden,et al.  Interval estimation of a global optimum for large combinatorial problems , 1979 .

[3]  G. S. Lueker,et al.  Asymptotic Methods in the Probabilistic Analysis of Sequencing and Packing Heuristics , 1988 .

[4]  Ravindra K. Ahuja,et al.  Use of Representative Operation Counts in Computational Testing of Algorithms , 1996, INFORMS J. Comput..

[5]  Rainer E. Burkard,et al.  The asymptotic probabilistic behaviour of quadratic sum assignment problems , 1983, Z. Oper. Research.

[6]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[7]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[8]  M. Braga,et al.  Exploratory Data Analysis , 2018, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[9]  Bruce L. Golden,et al.  Experimentation in optimization , 1986 .

[10]  Michael Pinedo,et al.  GUIDELINES FOR REPORTING COMPUTATIONAL RESULTS IN IIE TRANSACTIONS , 1993 .

[11]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[12]  N. Biggs.  THE TRAVELING SALESMAN PROBLEM A Guided Tour of Combinatorial Optimization , 1986 .

[13]  Reha Uzsoy,et al.  Decomposition Methods for Complex Factory Scheduling Problems , 1996 .

[14]  Alexander H. G. Rinnooy Kan,et al.  A stochastic method for global optimization , 1982, Math. Program..

[15]  Laura A. Sanchis,et al.  On the Complexity of Test Case Generation for NP-Hard Problems , 1990, Inf. Process. Lett..

[16]  Richard S. Barr,et al.  Feature Article - Reporting Computational Experiments with Parallel Algorithms: Issues, Measures, and Experts' Opinions , 1993, INFORMS J. Comput..

[17]  James M. Crawford,et al.  Experimental Results on the Crossover Point inSatis ability , 1993 .

[18]  Charles H. Reilly,et al.  The Effects of Coefficient Correlation Structure in Two-Dimensional Knapsack Problems on Solution Procedure Performance , 2000 .

[19]  Martin Grötschel,et al.  On the symmetric travelling salesman problem I: Inequalities , 1979, Math. Program..

[20]  Christopher Vyn Jones,et al.  Visualization and Optimization , 1997 .

[21]  Reha Uzsoy,et al.  Integrating Interval Estimates of Global Optima and Local Search Methods for Combinatorial Optimization Problems , 2000, J. Heuristics.

[22]  Eitan Zemel.  Measuring the Quality of Approximate Solutions to Zero-One Programming Problems , 1981, Math. Oper. Res..

[23]  Stelios H. Zanakis,et al.  A simulation study of some simple estimators for the three-parameter weibull distribution , 1979 .

[24]  Reha Uzsoy,et al.  Scheduling semiconductor test operations: Minimizing maximum lateness and number of tardy jobs on a single machine , 1992 .

[25]  Benjamin W. Lin,et al.  Controlled Experimental Design for Statistical Comparison of Integer Programming Algorithms , 1979 .

[26]  David J. Groggel,et al.  Practical Nonparametric Statistics , 2000, Technometrics.

[27]  Harvey J. Greenberg.  Computational Testing: Why, How and How Much , 1990, INFORMS J. Comput..

[28]  Mauricio G. C. Resende,et al.  Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[29]  J. King.  Machine-component grouping in production flow analysis: an approach using a rank order clustering algorithm , 1980 .

[30]  Jan Karel Lenstra,et al.  Sequencing by enumerative methods , 1977 .

[31]  Michael Pinedo,et al.  Scheduling: Theory, Algorithms, and Systems , 1994 .

[32]  Stelios H. Zanakis,et al.  A good simple percentile estimator of the weibull shape parameter for use when all three parameters are unknown , 1982 .

[33]  Pablo Moscato,et al.  On the Performance of Heuristics on Finite and Infinite Fractal Instances of the Euclidean Traveling Salesman Problem , 1998, INFORMS J. Comput..

[34]  Larry S. Yaeger,et al.  Visualization of natural phenomena , 1993 .

[35]  Ronald L. Rardin,et al.  Using a hybrid of exact and genetic algorithms to design survivable networks , 2002, Comput. Oper. Res..

[36]  Robert S. Garfinkel,et al.  The Bottleneck Traveling Salesman Problem: Algorithms and Probabilistic Analysis , 1978, JACM.

[37]  Ronald L. Rardin,et al.  Partial polyhedral description and generation of discrete optimization problems with known optima , 1992 .

[38]  Kenneth R. Baker,et al.  Minimizing maximum lateness with job families , 2000, Eur. J. Oper. Res..

[39]  Jeffrey L. Arthur,et al.  Generating Travelling-Salesman Problems with Known Optimal Tours , 1988 .

[40]  Reha Uzsoy,et al.  Benchmarks for shop scheduling problems , 1998, Eur. J. Oper. Res..

[41]  Hector J. Levesque,et al.  Hard and Easy Distributions of SAT Problems , 1992, AAAI.

[42]  Clyde L. Monma,et al.  On the Complexity of Scheduling with Batch Setup Times , 1989, Oper. Res..

[43]  Stephen G. Nash,et al.  Guidelines for reporting results of computational experiments. Report of the ad hoc committee , 1991, Math. Program..

[44]  Ulrich Derigs,et al.  Using Confidence Limits for the Global Optimum in Combinatorial Optimization , 1985, Oper. Res..

[45]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[46]  R. Fisher,et al.  Limiting forms of the frequency distribution of the largest or smallest member of a sample , 1928, Mathematical Proceedings of the Cambridge Philosophical Society.

[47]  Edward R. Tufte,et al.  Envisioning Information , 1990 .

[48]  Kenneth R. Baker.  Heuristic procedures for scheduling job families with setups and due dates , 1999 .

[49]  C. L. Moodie,et al.  A HEURISTIC METHOD OF ASSEMBLY LINE BALANCING FOR ASSUMPTIONS OF CONSTANTOR VARIABLE WORK ELEMENT TIMES , 1964 .

[50]  Eugene L. Lawler,et al.  The Traveling Salesman Problem: A Guided Tour of Combinatorial Optimization , 1985 .

[51]  Chris N. Potts,et al.  Single machine scheduling with batch set-up times to minimize maximum lateness , 1997, Ann. Oper. Res..

[52]  Kiyoshi Yoneda,et al.  OPTIMAL NUMBER OF DIGITS TO REPORT , 1996 .

[53]  Laura A. Sanchis,et al.  Generating Hard and Diverse Test Sets for NP-hard Graph Problems , 1995, Discret. Appl. Math..

[54]  Bruce L. Golden,et al.  Point estimation of a global optimum for large combinatorial problems , 1978 .

[55]  D. Dannenbring.  Procedures for Estimating Optimal Solution Values for Large Combinatorial Problems , 1977 .

[56]  Ronald L. Rardin,et al.  Test Problems for Computational Experiments -- Issues and Techniques , 1982 .

[57]  Laura A. Sanchis,et al.  Test Case Construction for the Vertex Cover Problem , 1992, Computational Support for Discrete Mathematics.

[58]  Panos M. Pardalos,et al.  A test problem generator for the Steiner problem in graphs , 1993, TOMS.

[59]  Richard M. Karp,et al.  Probabilistic Analysis of Partitioning Algorithms for the Traveling-Salesman Problem in the Plane , 1977, Math. Oper. Res..

[60]  John M. Mulvey,et al.  On Reporting Computational Experiments with Mathematical Software , 1979, TOMS.

[61]  W. Hopp,et al.  ECONOMIC PRODUCTION QUOTAS FOR PULL MANUFACTURING SYSTEMS , 1993 .

[62]  Brian W. Kernighan,et al.  An Effective Heuristic Algorithm for the Traveling-Salesman Problem , 1973, Oper. Res..

[63]  Edward R. Tufte,et al.  The Visual Display of Quantitative Information , 1986 .

[64]  Ronald L. Graham,et al.  Bounds on Multiprocessing Timing Anomalies , 1969, SIAM Journal of Applied Mathematics.

[65]  Balakrishnan Krishnamurthy,et al.  Constructing Test Cases for Partitioning Heuristics , 1987, IEEE Transactions on Computers.

[66]  James R. Evans,et al.  Heuristic methods and applications: A categorized survey , 1989 .

[67]  Bruce L. Golden,et al.  A statistical approach to the tsp , 1977, Networks.