Usefulness Evaluation of Simulation in Server System Testing

This paper evaluates the usefulness of different simulators developed in server system testing for a mobile application during 5 years, with the purpose of identifying the advantages and weaknesses of each simulation method in the given setup. The study found that simulation was used on almost every interface to external systems, and two factors that affected the usefulness of simulators over long term were the functionality supported and the applicable usage scenarios. Large-scale simulation did not keep yielding high returns. In the mobile application of server/client architecture, emulating external servers raised the testing capability, and emulating mobile clients led to higher efficiency. The findings in this study can be used as references for software teams to plan simulation in QA activities properly for faster QA cycles, deeper understanding of the system, more stable test results and lower resource costs.

[1]  Alan M. Christie.  Simulation: An Enabling Technology in Software Engineering , 1999 .

[2]  Satoshi Matsuoka,et al.  Overview of a performance evaluation system for global computing scheduling algorithms , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[3]  Jun Han,et al.  Scalable Emulation of Enterprise Systems , 2009, 2009 Australian Software Engineering Conference.

[4]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[5]  John H. Howard,et al.  A virtual machine emulator for performance evaluation , 1980, CACM.

[6]  Henri Casanova,et al.  Scheduling distributed applications: the SimGrid simulation framework , 2003, CCGrid 2003. 3rd IEEE/ACM International Symposium on Cluster Computing and the Grid, 2003. Proceedings..

[7]  Yan Ma,et al.  Automated Large-Scale Simulation Test-Data Generation for Object-Oriented Software Systems , 2007, The First International Symposium on Data, Privacy, and E-Commerce (ISDPE 2007).

[8]  Yinong Chen,et al.  Mutation-Based Simulation Test Data Generation for Testing Complex Real-Time Software , 2007, 40th Annual Simulation Symposium (ANSS'07).

[9]  Cyril Briquet,et al.  Reproducible testing of distributed software with middleware virtualization and simulation , 2008, PADTAD '08.

[10]  Joel E. Henry,et al.  Assessing and improving testing of real-time software using simulation , 2003, 36th Annual Simulation Symposium, 2003..

[11]  Osman Balci,et al.  Guidelines For Selecting And Using Simulation Model Verification Techniques , 1989, 1989 Winter Simulation Conference Proceedings.

[12]  Dietmar Pfahl,et al.  Simulation Methods , 2019, Introductory Econometrics for Finance.

[13]  Ying Jiang,et al.  QoS Consistency as Basis of Reputation Measurement of Web Service , 2007, The First International Symposium on Data, Privacy, and E-Commerce (ISDPE 2007).