Architectural characterization and similarity analysis of sunspider and Google's V8 Javascript benchmarks

Today, more than 99% of web-browsers are enabled with Javascript capabilities, and Javascript's popularity is only going to increase in the future. However, due to bytecode interpretation, Javascript codes suffer from severe performance penalty (up to 50x slower) compared to the corresponding native C/C++ code. We recognize that the first step to bridge this performance gap is to understand the the architectural execution characteristics of Javascript benchmarks. Therefore, this paper presents an in-depth architectural characterization of widely used V8 and Sunspider Javascript benchmarks using Google's V8 javascript engine. Using statistical data analysis techniques, our characterization study discovers and explains correlation among different execution characteristics in microarchitecture dependent as well as microarchitecture independent fashion. Furthermore, our study measures (dis)similarity among 33 different Javascript benchmarks and discusses its implications. Given the widespread use of Javascripts, we believe our findings are useful for both performance analysis and benchmarking communities.

[1]  Trevor N. Mudge,et al.  Analysis of branch prediction via data compression , 1996, ASPLOS VII.

[2]  R. Wisniewski,et al.  Performance and Environment Monitoring for Whole-System Characterization and Optimization , 2004 .

[3]  Mikko H. Lipasti,et al.  An architectural evaluation of Java TPC-W , 2001, Proceedings HPCA Seventh International Symposium on High-Performance Computer Architecture.

[4]  Lieven Eeckhout,et al.  Using cycle stacks to understand scaling bottlenecks in multi-threaded workloads , 2011, 2011 IEEE International Symposium on Workload Characterization (IISWC).

[5]  Ben Zorn,et al.  JSMeter: Measuring JavaScript Behavior in the Wild , 2010 .

[6]  Ramesh Radhakrishnan,et al.  Characterization of Java applications at bytecode and ultra-SPARC machine code levels , 1999, Proceedings 1999 IEEE International Conference on Computer Design: VLSI in Computers and Processors (Cat. No.99CB37040).

[7]  Yue Li,et al.  Workload characterization of biometric applications on Pentium 4 microarchitecture , 2005, IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium, 2005..

[8]  John Paul Shen,et al.  Scaling and characterizing database workloads: bridging the gap between research and practice , 2003, Proceedings. 36th Annual IEEE/ACM International Symposium on Microarchitecture, 2003. MICRO-36..

[9]  Lizy Kurian John,et al.  A Performance Counter Based Workload Characterization on Blue Gene/P , 2008, 2008 37th International Conference on Parallel Processing.

[10]  Lieven Eeckhout,et al.  Microarchitecture-Independent Workload Characterization , 2007, IEEE Micro.

[11]  Ian M. Steiner,et al.  Characterizing a Complex J2EE Workload: A Comprehensive Analysis and Opportunities for Optimizations , 2007, 2007 IEEE International Symposium on Performance Analysis of Systems & Software.

[12]  Luis Ceze,et al.  Checked Load: Architectural support for JavaScript type-checking on mobile processors , 2011, 2011 IEEE 17th International Symposium on High Performance Computer Architecture.

[13]  Narayanan Vijaykrishnan,et al.  Architectural issues in Java runtime systems , 2000, Proceedings Sixth International Symposium on High-Performance Computer Architecture. HPCA-6 (Cat. No.PR00550).

[14]  Lieven Eeckhout,et al.  Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites , 2005, IEEE International Symposium on Performance Analysis of Systems and Software, 2005. ISPASS 2005..

[15]  Lizy Kurian John,et al.  On the representativeness of embedded Java benchmarks , 2008, 2008 IEEE International Symposium on Workload Characterization.

[16]  Lieven Eeckhout,et al.  Comparing Benchmarks Using Key Microarchitecture-Independent Characteristics , 2006, 2006 IEEE International Symposium on Workload Characterization.

[17]  Lizy K. JohnLaboratory.  A Performance Study of Modern Web Applications #2 Performance Evaluation and Prediction , 1999 .

[18]  Srihari Makineni,et al.  Exploring the cache design space for large scale CMPs , 2005, CARN.

[19]  Lieven Eeckhout,et al.  Performance prediction based on inherent program similarity , 2006, 2006 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[20]  Lizy Kurian John,et al.  Analysis of redundancy and application balance in the SPEC CPU2006 benchmark suite , 2007, ISCA '07.

[21]  Ramesh Radhakrishnan,et al.  A Performance Study of Modern Web Server Applications , 1999, Euro-Par.

[22]  Narayanan Vijaykrishnan,et al.  Using complete system simulation to characterize SPECjvm98 benchmarks , 2000, ICS '00.

[23]  Lizy K. John,et al.  Execution characteristics of multimedia applications on a Pentium II processor , 2000, Conference Proceedings of the 2000 IEEE International Performance, Computing, and Communications Conference (Cat. No.00CH37086).

[24]  Ramesh Radhakrishnan,et al.  Execution characteristics of object oriented programs on the UltraSPARC-II , 1998, Proceedings. Fifth International Conference on High Performance Computing (Cat. No. 98EX238).

[25]  Brad Calder,et al.  Structures for phase classification , 2004, IEEE International Symposium on - ISPASS Performance Analysis of Systems and Software, 2004.

[26]  Matthias Hauswirth,et al.  Using Hardware Performance Monitors to Understand the Behavior of Java Applications , 2004, Virtual Machine Research and Technology Symposium.

[27]  Lizy Kurian John,et al.  Workload characterization of multithreaded java servers , 2001, 2001 IEEE International Symposium on Performance Analysis of Systems and Software. ISPASS..

[28]  Lieven Eeckhout,et al.  How java programs interact with virtual machines at the microarchitectural level , 2003, OOPSLA '03.

[29]  Matthias Hauswirth,et al.  Characterizing the design and performance of interactive java applications , 2010, 2010 IEEE International Symposium on Performance Analysis of Systems & Software (ISPASS).

[30]  Dileep Bhandarkar,et al.  Performance characterization of the Pentium Pro processor , 1997, Proceedings Third International Symposium on High-Performance Computer Architecture.

[31]  Kenneth C. Barr,et al.  Summarizing multiprocessor program execution with versatile, microarchitecture-independent snapshots , 2006 .

[32]  Scott A. Mahlke,et al.  Dynamic parallelization of JavaScript applications using an ultra-lightweight speculation mechanism , 2011, 2011 IEEE 17th International Symposium on High Performance Computer Architecture.

[33]  Xi Yang,et al.  Looking back on the language and hardware revolutions: measured power, performance, and scaling , 2011, ASPLOS XVI.

[34]  Matthias Hauswirth,et al.  Producing wrong data without doing anything obviously wrong! , 2009, ASPLOS.

[35]  Peter F. Sweeney,et al.  Understanding the cost of thread migration for multi-threaded Java applications running on a multicore platform , 2009, 2009 IEEE International Symposium on Performance Analysis of Systems and Software.

[36]  Luis Ceze,et al.  A limit study of JavaScript parallelism , 2010, IEEE International Symposium on Workload Characterization (IISWC'10).

[37]  Charles Yount,et al.  Characterization of SPEC CPU2006 and SPEC OMP2001: Regression Models and their Transferability , 2008, ISPASS 2008 - IEEE International Symposium on Performance Analysis of Systems and software.