Online GANs for Automatic Performance Testing

In this paper we present a novel algorithm for automatic performance testing that uses an online variant of the Generative Adversarial Network (GAN) to optimize the test generation process. The objective of the proposed approach is to generate, for a given test budget, a test suite containing a high number of tests revealing performance defects. This is achieved using a GAN to generate the tests and predict their outcome. This GAN is trained online while generating and executing the tests. The proposed approach does not require a prior training set or model of the system under test. We provide an initial evaluation the algorithm using an example test system, and compare the obtained results with other possible approaches.We consider that the presented algorithm serves as a proof of concept and we hope that it can spark a research discussion on the application of GANs to test generation.

[1]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[2]  Elaine J. Weyuker,et al.  Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study , 2000, IEEE Trans. Software Eng..

[3]  S. V. Subrahmanya,et al.  Object driven performance testing of Web applications , 2000, Proceedings First Asia-Pacific Conference on Quality Software.

[4]  Rommel G. Regis,et al.  A Survey of Surrogate Approaches for Expensive Constrained Black-Box Optimization , 2019, WCGO.

[5]  Johan Lilius,et al.  Scenario-based Testing of a Ship Collision Avoidance System , 2020, 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[6]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[7]  Tim Menzies,et al.  FLASH: A Faster Optimizer for SBSE Tasks , 2017, ArXiv.

[8]  Dragos Truscan,et al.  Using Deep Reinforcement Learning for Exploratory Performance Testing of Software Systems With Multi-Dimensional Input Spaces , 2020, IEEE Access.

[9]  Chee Peng Lim,et al.  Automatic design of hyper-heuristic based on reinforcement learning , 2018, Inf. Sci..

[10]  Rafael Prikladnicki,et al.  How Machine Learning Has Been Applied in Software Engineering? , 2020, ICEIS.

[11]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[12]  Michel Gendreau,et al.  Handbook of Metaheuristics , 2010 .

[13]  Dragos Truscan,et al.  Automatic exploratory performance testing using a discriminator neural network , 2020, 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[14]  Alexander Nareyek,et al.  Choosing search heuristics by non-stationary reinforcement learning , 2004 .