Benchmarking Domain Specific Processors: A Case Study of Evaluating a Smart Card Processor Design

Benchmarking domain specific processors, compared with benchmarking general-purpose computers, has its own particularity, since the performance of domain specific processors strongly depends on target applications. System architects often face the problem of lacking benchmarks that represent the application space of a specific embedded system. Furthermore, benchmarking is often hindered from offering a clear performance analysis due to the presence of complex software, like operating system and Java virtual machine. In this paper we demonstrate a case study of benchmarking a smart card processor designed for Java applications. The benchmarking approach involves selecting suitable benchmarks from a set of given applications and analyzing the benchmarking results based on an abstract performance model to identify performance bottlenecks at the level of complex operations, each of which consists of a sequence of processor instructions. It evaluates the hardware and software in isolation and provides reliable information for system enhancement.

[1]  H. Vandierendonck,et al.  Experiments with subsetting benchmark suites , 2004, IEEE International Workshop on Workload Characterization, 2004. WWC-7. 2004.

[2]  Lizy K. John,et al.  Performance prediction using program similarity , 2006 .

[3]  Margo I. Seltzer,et al.  HBench:Java: An application‐specific benchmarking framework for Java Virtual Machines , 2001, Concurr. Comput. Pract. Exp..

[4]  Margo I. Seltzer,et al.  HBench:Java: an application-specific benchmarking framework for Java virtual machines , 2000, JAVA '00.

[5]  Trevor Mudge,et al.  MiBench: A free, commercially representative embedded benchmark suite , 2001 .

[6]  Lieven Eeckhout,et al.  Quantifying the Impact of Input Data Sets on Program Behavior and its Applications , 2003, J. Instr. Level Parallelism.

[7]  Lieven Eeckhout,et al.  How java programs interact with virtual machines at the microarchitectural level , 2003, OOPSLA '03.

[8]  Lieven Eeckhout,et al.  Performance prediction based on inherent program similarity , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[9]  Douglas M. Hawkins,et al.  A statistically rigorous approach for improving simulation methodology , 2003, The Ninth International Symposium on High-Performance Computer Architecture, 2003. HPCA-9 2003. Proceedings..

[10]  Lieven Eeckhout,et al.  Measuring benchmark similarity using inherent program characteristics , 2006, IEEE Transactions on Computers.

[11]  Alan Jay Smith,et al.  Analysis of benchmark characteristics and benchmark performance prediction , 1996, TOCS.

[12]  Lieven Eeckhout,et al.  Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[13]  Zhiqun Chen,et al.  Java CardTM Technology for Smart Cards , 2007 .