SubsetTrio: An evolutionary, geometric, and statistical benchmark subsetting framework

Motivated by excessively high benchmarking efforts caused by a rapidly expanding design space, increasing system complexity, and prevailing practices based on ad-hoc and subjective schemes, this article seeks to enhance architecture exploration and evaluation efficiency by strategically integrating a genetic algorithm, 3-D geometrical rendering, and multivariate statistical analysis into one unified methodology framework—SubsetTrio—capable of subsetting any given benchmark suite based on its inherent workload characteristics, desired workload space coverage, and the total execution time intended by the user. By encoding both representativity (i.e., workload space coverage represented by the volume of the convex hull of benchmarks) and efficiency (i.e., total run time) as a co-optimization objective of a survival-of-the-fittest evolutionary algorithm, we can systematically determine a globally “fittest” (i.e., most representative and efficient) benchmark subset according to the workload space coverage threshold specified by the user. We demonstrate the usage, efficacy, and efficiency of the proposed technique by conducting a thorough case study on the SPEC benchmark suite, and evaluate its validity based on 50 commercial computer systems. Compared to the state-of-the-art statistical subsetting approach based on the Principal Component Analysis (PCA), SubsetTrio could select a significantly more time-efficient subset, while covering the same or higher workload space.