Managing Performance Testing with Release Certification and Data Correlation

Performance testing is a key element of industrial software development. While basic performance testing concepts are well understood, it is less clear how to manage performance tests in practice. I.e., we have encountered the following two problems. (1) While testing textbooks prescribe writing tests against performance goals, we find that it is impractical to gather from business analysts performance goals that are detailed enough for finding subtle performance bugs. (2) Once performance tests are conducted, we were asked questions such as the following, which we found hard to answer. How can you be confident that the executed tests assess the performance of the software truthfully? To enable practitioners to address these problems, this paper introduces two additional performance testing process components, which we call release certification and test data correlation. Our key idea to address problem (1) is to run two different versions of the same subject application side-by-side in the same test environment. This allows us to use the performance profile of the previous version as the detailed performance specification of the version under test. Our key idea to address the questions of (2) is to correlate the performance measurements of the test and production environments. We also report on our experience of applying this implementation on several releases of a commercial airline sales application.