Performance testing of combinatorial solvers with isomorph class instances

Combinatorial optimization problems expressed as Boolean constraint satisfaction problems (BCSPs) arise in several contexts, ranging from the classical unate set-packing problems to the binate minimum cover problems, including the Haplotype Inference by Pure Parsimony (HIPP) problem. These problems are being solved under different formulations and in different formats. Results of experiments that are reported can be seldom compared and replicated. This paper is not about 'the best BCSP solver'. Rather, it is a case study of how the scientific method can be applied to comparing the performance of not only BCSP solvers but also other solvers that address NP-hard problems. The approach is founded on two premises: (1) the introduction of instance isomorphs as families of equivalence classes, based on randomized replicas of a given reference instance, and (2) the use of isomorph classes for the design of reproducible experiments with BCSP solvers that includes performance testing hypotheses. We introduce a number of BCSP reference instances from different domains, generate isomorph classes and use various versions of cplex to characterize the solver performance and the isomorph classes themselves. This methodology may make it easier to (1) reliably improve the performance of combinatorial solvers and, (2) report results of experiments under the proposed schema.

[1]  E. Lloyd.  Statistical Theory and Methodology in Science and Engineering , 1961 .

[2]  J. Hannan,et al.  Introduction to probability and mathematical statistics , 1986 .

[3]  Matthias F. Stallmann,et al.  High-contrast algorithm behavior: observation, conjecture, and experimental design , 2007 .

[4]  John N. Hooker,et al.  Testing heuristics: We have it all wrong , 1995, J. Heuristics.

[5]  David S. Johnson,et al.  A theoretician's guide to the experimental analysis of algorithms , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[6]  Y. Zhu,et al.  Heuristics for a Brokering Set Packing Problem , 2004, AI&M.

[7]  Matthias F. Stallmann,et al.  On SAT instance classes and a method for reliable performance experiments with SAT solvers , 2005, Annals of Mathematics and Artificial Intelligence.

[8]  Yi Zhu,et al.  Heuristics for a bidding problem , 2006, Comput. Oper. Res..

[9]  Franc BrglezApril.  Design of Experiments to Evaluate CAD Algorithms: Which Improvements Are Due to Improved Heuristic and Which Are Merely Due to Chance? , 1998 .

[10]  S. Yang,et al.  Logic Synthesis and Optimization Benchmarks User Guide Version 3.0 , 1991 .

[11]  Luca Trevisan,et al.  The Approximability of Constraint Satisfaction Problems , 2001, SIAM J. Comput..

[12]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[13]  Mauricio G. C. Resende,et al.  Designing and reporting on computational experiments with heuristic methods , 1995, J. Heuristics.

[14]  D. Feitelson.  Experimental Computer Science: the Need for a Cultural Change , 2006 .

[15]  Matthias F. Stallmann,et al.  Effective bounding techniques for solving unate and binate covering problems , 2005, Proceedings. 42nd Design Automation Conference, 2005..

[16]  Catherine C. McGeoch.  Experimental analysis of algorithms , 1986 .

[17]  Fabio Somenzi,et al.  Logic synthesis and verification algorithms , 1996 .

[18]  L. N. Balaam,et al.  Statistical Theory and Methodology in Science and Engineering , 1966 .

[19]  Franc Brglez,et al.  Design of experiments and evaluation of BDD ordering heuristics , 2001, International Journal on Software Tools for Technology Transfer.

[20]  F. Brglez,et al.  Design of experiments in BDD variable ordering: lessons learned , 1998, 1998 IEEE/ACM International Conference on Computer-Aided Design. Digest of Technical Papers (IEEE Cat. No.98CB36287).

[21]  Thomas Stützle,et al.  Evaluating Las Vegas Algorithms: Pitfalls and Remedies , 1998, UAI.

[22]  Giovanni De Micheli,et al.  Synthesis and Optimization of Digital Circuits , 1994 .

[23]  W. R. Buckland,et al.  Statistical Theory and Methodology in Science and Engineering. , 1960 .

[24]  Cecilia R. Aragon,et al.  Optimization by Simulated Annealing: An Experimental Evaluation; Part II, Graph Coloring and Number Partitioning , 1991, Oper. Res..

[25]  References , 1971 .