An Industrial Case Study of Customizing Operational Profiles Using Log Compression

Large customers commonly request on-site capacity testing before upgrading to a new version of a mission critical telecom application. Customers fear that the new version cannot handle their current workload. These on-site engagements are costly and time consuming. These engagements prolong the upgrade cycle for products and reduce the revenue stream of rapidly growing companies. We present an industrial case study for a lightweight simple approach for customizing the operational profile for a particular deployment. The approach flags sequences of repeated events out of millions of events in execution logs. A performance engineer can identify noteworthy usage scenarios using these flagged sequences. The identified scenarios are used to customize the operational profile. Using a customized profile for performance testing alleviates customer's concerns about the performance of a new version of an application, and results in more realistic performance and reliability estimates. The simplicity of our approach ensures that customers can easily grasp the results of our analysis over other more complex analysis approaches. We demonstrate the feasibility and applicability of our approach by customizing the operational profile of an enterprise telecom application.

[1]  Wayne Citrin,et al.  Using formalized temporal message‐flow diagrams , 1995, Softw. Pract. Exp..

[2]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[3]  Bjørn N. Freeman-Benson,et al.  Visualizing dynamic software system information through high-level models , 1998, OOPSLA '98.

[4]  Elaine J. Weyuker,et al.  A metric for predicting the performance of an application under a growing workload , 2002, IBM Syst. J..

[5]  Abdelwahab Hamou-Lhadj,et al.  Measuring various properties of execution traces to help build better trace analysis tools , 2005, 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'05).

[6]  Virgílio A. F. Almeida,et al.  A methodology for workload characterization of E-commerce sites , 1999, EC '99.

[7]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[8]  Spencer Rugaber,et al.  Using visualization for architectural localization and extraction , 1997, Proceedings of the Fourth Working Conference on Reverse Engineering.

[9]  Virgílio A. F. Almeida,et al.  Performance by Design - Computer Capacity Planning By Example , 2004 .

[10]  John D. Musa,et al.  The operational profile , 1996 .

[11]  Elaine J. Weyuker,et al.  Software performance testing based on workload characterization , 2002, WOSP '02.

[12]  Jeffrey Voas,et al.  Will the Real Operational Profile Please Stand Up , 2000 .

[13]  Nevil Brownlee,et al.  Network Event Detection with T-Entropy , 2005 .

[14]  Abdelwahab Hamou-Lhadj,et al.  A survey of trace exploration tools and techniques , 2004, CASCON.

[15]  Rainer Koschke,et al.  Locating Features in Source Code , 2003, IEEE Trans. Software Eng..

[16]  A. L. Gilbert,et al.  Documentation in a software maintenance environment , 1988, Proceedings. Conference on Software Maintenance, 1988..

[17]  Kai Koskimies,et al.  SCED: A TOOL FOR DYNAMIC MODELLING OF OBJECT SYSTEMS , 1996 .

[18]  Adrian Kuhn,et al.  Exploiting the Analogy Between Traces and Signal Processing , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[19]  Jeffrey M. Voas Quality Time - Will the Real Operational Profile Please Stand Up? , 2000, IEEE Softw..

[20]  Nick Mitchell,et al.  Visualizing the Execution of Java Programs , 2001, Software Visualization.

[21]  Mariacarla Calzarossa,et al.  Models of mail server workloads , 2001, Perform. Evaluation.

[22]  Elaine J. Weyuker,et al.  Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study , 2000, IEEE Trans. Software Eng..