Overview of Popular Benchmark Sets

One of the most difficult tasks CAD users face is the evaluation and comparison of different tools and algorithms. For commercial software purchasers, it is vital to understand how well a given tool does the required job and which of many possible choices is best for the kinds of problems a user will face. For the tool developer, whether in academia or industry, the efficiency of critical algorithms must be measured and compared to understand both tool behavior and progress over time. Over the years, there have been many attempts to create and use neutral benchmarks for tool evaluation and comparison. Typically, a benchmark set consists of a collection of circuits in a common format, which attempt to represent a range of problems for evaluating algorithms and tools within an important problem domain. In principle, if everyone uses the same test cases to evaluate similar tools, it should be straightforward to compare results-although this is rarely true in reality. Several benchmark sets are widely used. The paper summarises briefly some of the more important ones. In addition, advanced research is under way to develop synthetic benchmarks with carefully controlled properties, with the objective of studying scaling and perturbation effect.