Performance analysts commonly use trace-based data containing hardware and software metrics to understand performance. The trace data is generated by instrumenting the code to increment a counter when an event occurs and to collect hardware and software metrics in a trace. Unfortunately, the act of collecting a trace can perturb the behavior that the trace is frying to capture. In this paper, we gain an understanding of perturbation due to measurement instrumentation of the system. We identify two mechanisms to quantify perturbation: inner and outer perturbation. Using inner perturbation, a performance analyst can determine when a run is perturbed by collecting too much information. Using outer perturbation, the performance analyst can determine if she can use the data from multiple runs as if the data were all from a single run. Our evaluation of these mechanisms lead to two results. First, we are surprised to find that even with minimal instrumentation overhead, which increased instructions executed by less than 3%, high perturbation resulted, which prevented one from correctly reasoning about metrics within a trace or across traces. Second, the instrumentation of different software metrics interact in subtle, and not always obvious, ways making the impact of instrumentation on perturbation difficult, if not impossible, to predict. Finally, we outline a methodology for collecting data while avoiding perturbation. When inner perturbation occurs, the performance analyst can spread out the data collection over multiple runs. When outer perturbation occurs, she can try different strategies for spreading out the data collection over multiple runs.
[1]
Allen D. Malony,et al.
Event-based performance perturbation: a case study
,
1991,
PPOPP '91.
[2]
Allen D. Malony,et al.
Models for performance perturbation analysis
,
1991,
PADD '91.
[3]
Matthias Hauswirth,et al.
Vertical profiling: understanding the behavior of object-priented applications
,
2004,
OOPSLA.
[4]
Chun Xia,et al.
Low Perturbation Address Trace Collection for Operating System, Multiprogrammed, and Parallel Workloads in Multiprocessors 1
,
1996
.
[5]
Allen D. Malony,et al.
Overhead Compensation in Performance Profiling
,
2004,
Parallel Process. Lett..
[6]
George C. Necula,et al.
CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs
,
2002,
CC.
[7]
Jack J. Dongarra,et al.
A Portable Programming Interface for Performance Evaluation on Modern Processors
,
2000,
Int. J. High Perform. Comput. Appl..
[8]
Donald J. Berndt,et al.
Using Dynamic Time Warping to Find Patterns in Time Series
,
1994,
KDD Workshop.
[9]
Harish Patil,et al.
Pin: building customized program analysis tools with dynamic instrumentation
,
2005,
PLDI '05.
[10]
Matthias Hauswirth,et al.
Aligning traces for performance evaluation
,
2006,
Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.
[11]
Matthias Hauswirth,et al.
Automating vertical profiling
,
2005,
OOPSLA '05.
[12]
Allen D. Malony,et al.
Perturbation analysis of high level instrumentation for SPMD programs
,
1993,
PPOPP '93.