Randomizing regression tests using game theory

As software evolves, the number of test-cases in the regression test suites continues to increase, requiring testers to prioritize their execution. Usually only a subset of the test cases is executed due to limited testing resources. This subset is often known to the developers who may try to “game” the system by committing insufficiently tested code for parts of the software that will not be tested. In this new ideas paper, we propose a novel approach for randomizing regression test scheduling, based on Stackelberg games for deployment of scarce resources. We apply this approach to randomizing test cases in such a way as to maximize the testers' expected payoff when executing the test cases. Our approach accounts for resource limitations (e.g., number of testers) and provides a probabilistic distribution for scheduling test cases. We provide an example application of our approach showcasing the idea of using Stackelberg games for randomized regression test scheduling.

[1]  R. Powell.  Defending against Terrorist Attacks with Limited Resources , 2007, American Political Science Review.

[2]  Mary Jean Harrold,et al.  Using random test selection to gain confidence in modified software , 2008, 2008 IEEE International Conference on Software Maintenance.

[3]  Lmg Loe Feijs,et al.  Prisoner's dilemma in software testing , 2001 .

[4]  Joseph Robert Horgan,et al.  A study of effective regression testing in practice , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[5]  Manish Jain,et al.  Computing optimal randomized resource allocations for massive security games , 2009, AAMAS.

[6]  J. Crisp,et al.  The Delphi method? , 1997, Nursing research.

[7]  Manish Jain,et al.  Computing optimal randomized resource allocations for massive security games , 2009, AAMAS 2009.

[8]  Sjouke Mauw,et al.  7e Nederlandse testdag, Eindhoven, 8 November 2001 : proceedings : Dutch testing day : 7th, Eindhoven, November 8, 2001 : proceedings , 2001 .

[9]  Stefan Biffl,et al.  Value-Based Management of Software Testing , 2006, Value-Based Software Engineering.

[10]  Milind Tambe,et al.  Security and Game Theory: Evaluating Deployed Decision-Support Systems for Security: Challenges, Analysis, and Approaches , 2011 .

[11]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[12]  Barry W. Boehm,et al.  Selecting an appropriate framework for value-based requirements prioritization , 2012, 2012 20th IEEE International Requirements Engineering Conference (RE).

[13]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[14]  Heinrich von Stackelberg.  Market Structure and Equilibrium , 2010 .

[15]  T. Başar,et al.  Dynamic Noncooperative Game Theory , 1982 .

[16]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[17]  Barry Boehm,et al.  Selecting the Most Appropriate Framework for Value Based Requirements Prioritization A Case Study , 2012 .

[18]  Milind Tambe,et al.  Security and Game Theory - Algorithms, Deployed Systems, Lessons Learned , 2011 .

[19]  A. Haurie,et al.  Sequential Stackelberg equilibria in two-person games , 1985 .

[20]  E. Rowland.  Theory of Games and Economic Behavior , 1946, Nature.

[21]  W. E. Fann.  Prisoner's Dilemma: John von Neumann, Game Theory, and the Puzzle of the Bomb , 1993 .

[22]  Rory V. O'Connor,et al.  Maximizing the value of the software development process by game theoretic analysis , 2010, PROFES '10.