THeME: a system for testing by hardware monitoring events

The overhead of test coverage analysis is dominated by monitoring the application, which is traditionally performed using instrumentation. However, instrumentation can prohibitively increase the time and especially the memory overhead of an application. As an alternative to instrumentation, we explore how recent hardware advances can be leveraged to improve the overheads of test coverage analysis. These hardware advances include hardware performance monitors and multicore technology. In this work, we present our system, THeME, a testing framework that replaces instrumentation with hardware monitoring. THeME consists of a runtime system that takes advantage of hardware mechanisms and multiple cores and a static component to further extend the coverage derived from hardware event sampling. The results show that up to 90% of the actual coverage can be determined with less time overhead and negligible code growth compared to instrumentation.

[1]  Amitabh Srivastava,et al.  Effectively prioritizing tests in development environment , 2002, ISSTA '02.

[2]  Vijay Janapa Reddi,et al.  Code coverage testing using hardware performance monitoring support , 2005, AADEBUG'05.

[3]  Mary Lou Soffa,et al.  Contention aware execution: online contention detection and response , 2010, CGO '10.

[4]  Chris Lattner,et al.  LLVM: AN INFRASTRUCTURE FOR MULTI-STAGE OPTIMIZATION , 2000 .

[5]  Adam A. Porter,et al.  Combining hardware and software instrumentation to classify program executions , 2010, FSE '10.

[6]  Jason Mars,et al.  Scenario Based Optimization: A Framework for Statically Enabling Online Optimizations , 2009, 2009 International Symposium on Code Generation and Optimization.

[7]  Wenguang Chen,et al.  RACEZ: a lightweight and non-invasive race detection tool for production applications , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[8]  Michael I. Jordan,et al.  Bug isolation via remote program sampling , 2003, PLDI.

[9]  Wei-Chung Hsu,et al.  Dynamic trace selection using performance monitoring hardware sampling , 2003, International Symposium on Code Generation and Optimization, 2003. CGO 2003..

[10]  Wei-Tek Tsai,et al.  Regression testing in an industrial environment , 1998, CACM.

[11]  Wenguang Chen,et al.  Taming hardware event samples for FDO compilation , 2010, CGO '10.

[12]  Dehao Chen,et al.  Feedback-Directed Optimizations in GCC with Estimated Edge Profiles from Hardware Event Sampling , 2008 .

[13]  Michael Stumm,et al.  Online performance analysis by statistical sampling of microprocessor performance counters , 2005, ICS '05.

[14]  James R. Larus,et al.  Exploiting hardware performance counters with flow and context sensitive profiling , 1997, PLDI '97.

[15]  Robert Hundt,et al.  Feedback-Directed Optimizations with Estimated Edge Profiles from Hardware Event Sampling , 2008 .

[16]  David S. Rosenblum,et al.  TESTTUBE: a system for selective regression testing , 1994, Proceedings of 16th International Conference on Software Engineering.

[17]  Michael F. P. O'Boyle,et al.  Rapidly Selecting Good Compiler Optimizations using Performance Counters , 2007, International Symposium on Code Generation and Optimization (CGO'07).

[18]  Mary Lou Soffa,et al.  Exploiting hardware advances for software testing and debugging: NIER track , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[19]  Matthew Arnold,et al.  A framework for reducing the cost of instrumented code , 2001, PLDI '01.

[20]  Raúl A. Santelices,et al.  Efficiently monitoring data-flow test coverage , 2007, ASE.

[21]  Michael R. Smith,et al.  A Hardware-Assisted Tool for Fast, Full Code Coverage Analysis , 2008, 2008 19th International Symposium on Software Reliability Engineering (ISSRE).

[22]  David A. Wood,et al.  Cache profiling and the SPEC benchmarks: a case study , 1994, Computer.

[23]  Michael I. Jordan,et al.  Scalable statistical bug isolation , 2005, PLDI '05.

[24]  Mary Lou Soffa,et al.  Characterizing multi-threaded applications based on shared-resource contention , 2011, (IEEE ISPASS) IEEE INTERNATIONAL SYMPOSIUM ON PERFORMANCE ANALYSIS OF SYSTEMS AND SOFTWARE.

[25]  M.L. Soffa,et al.  Demand-driven structural testing with dynamic instrumentation , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[26]  Jack J. Dongarra,et al.  A Scalable Cross-Platform Infrastructure for Application Performance Tuning Using Hardware Counters , 2000, ACM/IEEE SC 2000 Conference (SC'00).