A performance test design method and its implementation patterns for multi-services systems

Over the past years, the scope of telecommunication services has increased dramatically making network infrastructure-related services a very competitive market. Additionally, the traditional telecoms are combined with Internet technologies for providing a larger range of services. The obvious outcome is the increase of the number of subscribers and services demand. Due to this complexity, the performance testing of continuously evolving telecommunication services has become a real challenge and requires efficient and more powerful testing solutions. This ability highly depends on the performance test design and on the efficient use of hardware resources for test execution. This paper proposes a performance testing methodology which copes with the characteristics mentioned above. The methodology consists of a set of methods and patterns, exemplified for the TTCN-3 language, to realize adequate performance tests for multi-service systems. The effectiveness of this methodology is demonstrated throughout a case study on IP Multimedia Subsystem (IMS) performance testing.

[1]  Ken Jones,et al.  Practical programming in Tcl & Tk , 1995 .

[2]  Markus Lepper,et al.  TUB-TCI An Architecture for Dynamic Deployment of Test Components , 2003, TestCom.

[3]  W. Richard Stevens,et al.  Unix network programming , 1990, CCRV.

[4]  Debra J. Richardson,et al.  An evaluation of software test environment architectures , 1996, Proceedings of IEEE 18th International Conference on Software Engineering.

[5]  Cem Kaner,et al.  Testing Computer Software, Second Edition , 1993 .

[6]  Alberto Avritzer,et al.  Load testing software using deterministic state testing , 1993, ISSTA '93.

[7]  S. Winkel.  Optimal versus Heuristic Global Code Scheduling , 2007, 40th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO 2007).

[8]  Jack Shirazi.  Java Performance Tuning , 2000 .

[9]  Brent B Welch,et al.  Practical Programming in Tcl and Tk , 1999 .

[10]  Tibor Szabó,et al.  BCMP Performance Test with TTCN-3 Mobile Node Emulator , 2004, TestCom.

[11]  Stephen Gilmore,et al.  Software performance modelling using PEPA nets , 2004, WOSP '04.

[12]  R. E. Filman.  Internet computing , 2005 .

[13]  Said Elnaffar,et al.  Characterizing Computer Systems' Workloads , 2002 .

[14]  C. M. Sperberg-McQueen,et al.  Extensible markup language , 1997 .

[15]  Giovanni Denaro,et al.  Early performance testing of distributed software applications , 2004, WOSP '04.

[16]  Ina Schieferdecker,et al.  PerfTTCN, a TTCN language extension for performance testing , 1997 .

[17]  Ali R. Hurson,et al.  Scheduling and Load Balancing in Parallel and Distributed Systems , 1995 .

[18]  Virgílio A. F. Almeida,et al.  A methodology for workload characterization of E-commerce sites , 1999, EC '99.

[19]  S. Mohan,et al.  Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software [Book Review] , 2003, IEEE Software.

[20]  Douglas C. Schmidt,et al.  Pattern-Oriented Software Architecture, Patterns for Concurrent and Networked Objects , 2013 .

[21]  Jens Grabowski,et al.  TTCN-3 - A new Test Specification Language for Black-Box Testing of Distributed Systems , 2000 .

[22]  D. Ridley,et al.  Cape Town, South Africa , 1986, Journal of clinical ultrasound : JCU.

[23]  Elaine J. Weyuker,et al.  Software performance testing based on workload characterization , 2002, WOSP '02.

[24]  Yanyan Wang,et al.  Automating experimentation on distributed testbeds , 2005, ASE.

[25]  Michael Hackett,et al.  Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems , 2003 .

[26]  Janne Peisa,et al.  IMS Multimedia Telephony over Cellular Systems , 2007 .

[27]  Perry Cheng,et al.  Myths and realities: the performance impact of garbage collection , 2004, SIGMETRICS '04/Performance '04.

[28]  Kang Zhang,et al.  A Distributed Parallel Programming Framework , 2002, IEEE Trans. Software Eng..

[29]  Connie U. Smith,et al.  New Book - Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software , 2001, Int. CMG Conference.

[30]  Stephen Albin.  The Art of Software Architecture: Design Methods and Techniques , 2003 .

[31]  Douglas C. Schmidt,et al.  Patterns for concurrent and networked objects , 2000 .

[32]  Bernd Baumgarten,et al.  OSI Conformance Testing Methodology and TTCN , 1994 .

[33]  Varsha Apte,et al.  AutoPerf: an automated load generator and performance measurement tool for multi-tier software systems , 2007, WWW '07.

[34]  Roger Afima Ackerley.  Telecommunications performance engineering , 2004 .

[35]  Measuring telecommunication infrastructure needs and demand , 2000 .

[36]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[37]  Ina Schieferdecker,et al.  A Workload Model for Benchmarking BPEL Engines , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[38]  Giuseppe A. Di Lucca,et al.  Testing Web-based applications: The state of the art and future trends , 2006, Inf. Softw. Technol..

[39]  Ina Schieferdecker,et al.  Realizing Distributed TTCN-3 Test Systems with TCI , 2003, TestCom.

[40]  Henning Schulzrinne,et al.  SIPstone: Benchmarking SIP Server Performance , 2002 .

[41]  N. Stankovic,et al.  Patterns and Tools for Performance Testing , 2006, 2006 IEEE International Conference on Electro/Information Technology.

[42]  Steven Lester Halter,et al.  Enterprise Java Performance , 2000 .

[43]  Brian A. Nixon,et al.  Management of Performance Requirements for Information Systems , 2000, IEEE Trans. Software Eng..

[44]  Ina Schieferdecker,et al.  Test Patterns with TTCN-3 , 2004, FATES.

[45]  Ina Schieferdecker,et al.  Test Architectures for Distributed Systems: State of the Art and Beyond , 1998, IWTCS.

[46]  Steven Splaine,et al.  The web testing handbook , 2001 .

[47]  Ina Schieferdecker,et al.  A Workload Model for Benchmarking IMS Core Networks , 2007, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.

[48]  Gonzalo Camarillo,et al.  The 3G IP Multimedia Subsystem (IMS) , 2008 .

[49]  Thomas Magedanz,et al.  Design and Implementation of an Open IMS Core , 2005, MATA.

[50]  Douglas Lea.  Concurrent Programming in Java(TM): Design Principles and Patterns (3rd Edition) (Java (Addison-Wesley)) , 2006 .

[51]  Emily Halili,et al.  Apache JMeter , 2008 .

[52]  Elaine J. Weyuker,et al.  Generating test suites for software load testing , 1994, ISSTA '94.

[53]  John McLean,et al.  The STCL test tools architecture , 2002, IBM Syst. J..

[54]  Alan Jay Smith.  Workloads (creation and use) , 2007, CACM.

[55]  Wen Jian,et al.  TTCN-3:A New Test Specification Language for Black-Box Testing of Distributed Systems , 2001 .

[56]  Helmut Wolfram Neukirchen,et al.  Languages, tools and patterns for the specification of distributed real-time tests , 2004 .

[57]  Bernd Mohr,et al.  Evaluating OpenMP Performance Analysis Tools with the APART Test Suite , 2004, Euro-Par.

[58]  Joseph Y.-T. Leung,et al.  Handbook of Scheduling: Algorithms, Models, and Performance Analysis , 2004 .

[59]  Gonzalo Camarillo,et al.  The 3G IP Multimedia Subsystem (IMS): Merging the Internet and the Cellular Worlds, Second Edition , 2006 .

[60]  Abraham Silberschatz,et al.  Operating systems concepts with Java, 6th Edition , 2004 .

[61]  Max Jacobson,et al.  A Pattern Language: Towns, Buildings, Construction , 1981 .

[62]  Dieter Hogrefe,et al.  An introduction to the testing and test control notation (TTCN-3) , 2003, Comput. Networks.

[63]  梅村 恭司.  Andrew S.Tanenbaum 著, "Operating systems, Design and implementation", PRENTICE-HALL, INC., Englewood Cliffs, B5変形判, 719p., \4,120 , 1988 .

[64]  Ralf Kreher,et al.  UMTS Signaling: UMTS Interfaces, Protocols, Message Flows and Procedures Analyzed and Explained, Second Edition , 2005 .

[65]  Ina Schieferdecker,et al.  Distributed Load Tests with TTCN-3 , 2006, TestCom.

[66]  Claudia Klüppelberg,et al.  Telecommunication traffic, queueing models, and subexponential distributions , 1999, Queueing Syst. Theory Appl..

[67]  Daniel A. Menascé.  Load Testing, Benchmarking, and Application Performance Management for the Web , 2002, Int. CMG Conference.

[68]  Ina Schieferdecker,et al.  Distributed functional and load tests for Web services , 2004, International Journal on Software Tools for Technology Transfer.

[69]  Paul Amaranth.  A Tcl-based Multithreaded Test Harness , 1998, Tcl/Tk Conference.

[70]  Elaine J. Weyuker,et al.  Performance testing of software systems , 1998, WOSP '98.

[71]  Ahmet Can Keskin,et al.  Tcl/Tk: a strong basis for complex load testing systems , 2000 .

[72]  Virgílio A. F. Almeida,et al.  Capacity Planning for Web Performance: Metrics, Models, and Methods , 1998 .

[73]  Anders P. Ravn,et al.  Developing a TTCN-3 test harness for legacy software , 2006, AST '06.

[74]  David McDysan,et al.  ATM and MPLS Theory and Application: Foundations of Multi-Service Networking , 2002 .

[75]  Sean McDonald,et al.  Software Test Automation , 1999 .

[76]  Franz-Josef Banet,et al.  Flexible Universal Networks—A New Approach to Telecommunication Services? , 2004, Wirel. Pers. Commun..

[77]  Javier Romero,et al.  GSM, Gprs and Edge Performance , 2003 .

[78]  Daniel A. Menascé,et al.  TPC-W: A Benchmark for E-Commerce , 2002, IEEE Internet Comput..

[79]  Ye Wu,et al.  Testing and Quality Assurance for Component-Based Software , 2003 .

[80]  Abraham Silberschatz,et al.  Operating System Concepts with Java , 2006 .

[81]  Winfried Dulz,et al.  Automated TTCN-3 test case generation by means of UML sequence diagrams and Markov chains , 2003, 2003 Test Symposium.

[82]  Lieven Eeckhout,et al.  Performance Evaluation and Benchmarking , 2005 .

[83]  Ning Jiang,et al.  Designing a test suite for empirically-based middleware performance prediction , 2002 .

[84]  Stephan Schulz,et al.  Implementation of TTCN-3 Test Systems using the TRI , 2002, TestCom.

[85]  Elaine J. Weyuker,et al.  Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study , 2000, IEEE Trans. Software Eng..

[86]  Wolfgang Fischer,et al.  The Markov-Modulated Poisson Process (MMPP) Cookbook , 1993, Perform. Evaluation.

[87]  A. Hoffmann,et al.  Performance testing of a TINA platform , 1999, TINA '99. 1999 Telecommunications Information Networking Architecture Conference Proceedings (Cat. No.99EX368).

[88]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[89]  Liming Zhu,et al.  Model driven benchmark generation for web services , 2006, SOSE '06.

[90]  Preben E. Mogensen,et al.  Subjective evaluation of packet service performance in UMTS and heterogeneous networks , 2006, Q2SWinet '06.

[91]  Nikolas V. Stylianides,et al.  Signaling Performance Trials and Evaluation Results on a GPRS Platform , 2005, Inf. Syst. Frontiers.

[92]  Claude Caci,et al.  Testing object-oriented systems , 2000, SOEN.

[93]  K. Sreenivasan,et al.  On the construction of a representative synthetic workload , 1974, CACM.

[94]  Lawrence Harte.  IPTV Basics, Technology, Operation and Services , 2006 .

[95]  Paul Barford,et al.  Self-configuring network traffic generation , 2004, IMC '04.

[96]  T.J.O. Afullo.  Quality of service in telecommunications - the customer's perspective , 2004, 2004 IEEE Africon. 7th Africon Conference in Africa (IEEE Cat. No.04CH37590).