GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards

It has often been claimed that SBSE uses so-called ‘embarrassingly parallel’ algorithms that will imbue SBSE applications with easy routes to dramatic performance improvements. However, despite recent advances in multicore computation, this claim remains largely theoretical; there are few reports of performance improvements using multicore SBSE. This paper shows how inexpensive General Purpose computing on Graphical Processing Units (GPGPU) can be used to massively parallelise suitably adapted SBSE algorithms, thereby making progress towards cheap, easy and useful SBSE parallelism. The paper presents results for three different algorithms: NSGA2, SPEA2, and the Two Archive Evolutionary Algorithm, all three of which are adapted for multi-objective regression test selection and minimization. The results show that all three algorithms achieved performance improvements up to 25 times, using widely available standard GPUs. We also found that the speed-up was observed to be statistically strongly correlated to the size of the problem instance; as the problem gets harder the performance improvements also get better.

[1]  T. Y. Chen,et al.  Heuristics Towards The Optimization Of TheSize Of A Test Suite , 1970 .

[2]  Yuanyuan Zhang,et al.  Comparing the performance of metaheuristics for the analysis of multi-stakeholder tradeoffs in requirements optimisation , 2011, Inf. Softw. Technol..

[3]  Timothy G. Mattson,et al.  OpenCL Programming Guide , 2011 .

[4]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[5]  Jens H. Krüger,et al.  A Survey of General‐Purpose Computation on Graphics Hardware , 2007, Eurographics.

[6]  TorkarRichard,et al.  A systematic review of search-based testing for non-functional system properties , 2009 .

[7]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[8]  Enrique Alba,et al.  The jMetal framework for multi-objective optimization: Design and architecture , 2010, IEEE Congress on Evolutionary Computation.

[9]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[10]  Mark Harman,et al.  Highly Scalable Multi Objective Test Suite Minimisation Using Graphics Cards , 2011, SSBSE.

[11]  Xin Yao,et al.  A New Multi-objective Evolutionary Optimisation Algorithm: The Two-Archive Algorithm , 2006, 2006 International Conference on Computational Intelligence and Security.

[12]  Mark Harman,et al.  Regression testing minimization, selection and prioritization: a survey , 2012, Softw. Test. Verification Reliab..

[13]  Mark Harman,et al.  Faster Fault Finding at Google Using Multi Objective Regression Test Optimisation , 2011 .

[14]  Mark Harman,et al.  A post-placement side-effect removal algorithm , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[15]  Michael Potter,et al.  Adoption of computer aided software engineering (CASE) technology: an innovation adoption perspective , 1995, DATB.

[16]  Walter F. Tichy,et al.  Proceedings 25th International Conference on Software Engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[17]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[18]  Nicholas Nethercote,et al.  Valgrind: A Program Supervision Framework , 2003, RV@CAV.

[19]  Giuliano Antoniol,et al.  Concept Location with Genetic Algorithms: A Comparison of Four Distributed Architectures , 2010, 2nd International Symposium on Search Based Software Engineering.

[20]  Mark Harman,et al.  A multiple hill climbing approach to software module clustering , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[21]  John A. Clark,et al.  Formulating software engineering as a search problem , 2003, IEE Proc. Softw..

[22]  Adam A. Porter,et al.  A history-based test prioritization technique for regression testing in resource constrained environments , 2002, ICSE '02.

[23]  Outi Räihä,et al.  A survey on search-based software design , 2010, Comput. Sci. Rev..

[24]  S. Elbaum,et al.  The impact of test suite granularity on the cost-effectiveness of regression testing , 2002, Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.

[25]  Shigeyoshi Tsutsui,et al.  Solving quadratic assignment problems by genetic algorithms with GPU computation: a case study , 2009, GECCO '09.

[26]  Mark Harman,et al.  Test data regeneration: generating new test data from existing test data , 2012, Softw. Test. Verification Reliab..

[27]  Spiros Mancoridis,et al.  An architecture for distributing the computation of software clustering algorithms , 2001, Proceedings Working IEEE/IFIP Conference on Software Architecture.

[28]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.

[29]  Emanuel Melachrinoudis,et al.  Bi-criteria models for all-uses test suite reduction , 2004, Proceedings. 26th International Conference on Software Engineering.

[30]  William B. Langdon,et al.  A SIMD Interpreter for Genetic Programming on GPU Graphics Cards , 2007, EuroGP.

[31]  Kar Yan Tam,et al.  Factors Affecting the Adoption of Open Systems: An Exploratory Study , 1997, MIS Q..

[32]  James R. Cordy,et al.  Comprehending reality - practical barriers to industrial adoption of software maintenance automation , 2003, 11th IEEE International Workshop on Program Comprehension, 2003..

[33]  Martin J. Shepperd,et al.  Search Heuristics, Case-based Reasoning And Software Project Effort Prediction , 2002, GECCO.

[34]  George Mason,et al.  Procedures for Reducing the Size of Coverage-based Test Sets , 1995 .

[35]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[36]  Per Runeson,et al.  An Empirical Evaluation of Regression Testing Based on Fix-Cache Recommendations , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[37]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[38]  Günther Ruhe,et al.  Bi-objective release planning for evolving software systems , 2007, ESEC-FSE '07.

[39]  Kevin Skadron,et al.  Accelerating leukocyte tracking using CUDA: A case study in leveraging manycore coprocessors , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[40]  Tsong Yueh Chen,et al.  Dividing Strategies for the Optimization of a Test Suite , 1996, Inf. Process. Lett..

[41]  Shin Yoo,et al.  Search based data sensitivity analysis applied to requirement engineering , 2009, GECCO.

[42]  Wolfgang Banzhaf,et al.  Deployment of CPU and GPU-based genetic programming on heterogeneous devices , 2009, GECCO '09.

[43]  Mark Harman,et al.  Search Algorithms for Regression Test Case Prioritization , 2007, IEEE Transactions on Software Engineering.

[44]  A. Orso,et al.  Retesting software during development and maintenance , 2008, 2008 Frontiers of Software Maintenance.

[45]  Gregg Rothermel,et al.  Empirical studies of test‐suite reduction , 2002, Softw. Test. Verification Reliab..

[46]  Joseph Robert Horgan,et al.  Test set size minimization and fault detection effectiveness: a case study in a space application , 1997, Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97).

[47]  Gregg Rothermel,et al.  An empirical study of the effects of minimization on the fault detection capabilities of test suites , 1998, Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272).

[48]  Mark Kambites,et al.  Towards OpenMP for Java , 2004 .

[49]  Per Runeson,et al.  A systematic review on regression test selection techniques , 2010, Inf. Softw. Technol..

[50]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[51]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[52]  Rajiv Gupta,et al.  A methodology for controlling the size of a test suite , 1990, Proceedings. Conference on Software Maintenance 1990.

[53]  W. Eric Wong,et al.  Effect of test set minimization on fault detection effectiveness , 1998 .

[54]  Jerffeson Teixeira de Souza,et al.  The Human Competitiveness of Search Based Software Engineering , 2010, 2nd International Symposium on Search Based Software Engineering.

[55]  Mark Harman,et al.  Reformulating software engineering as a search problem , 2003 .

[56]  Mark Harman Making the Case for MORTO: Multi Objective Regression Test Optimization , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[57]  Man Leung Wong,et al.  Parallel multi-objective evolutionary algorithms on graphics processing units , 2009, GECCO '09.

[58]  Dinesh Manocha,et al.  GPUTeraSort: high performance graphics co-processor sorting for large database management , 2006, SIGMOD Conference.

[59]  Mark Harman,et al.  Measuring and Improving Latency to Avoid Test Suite Wear Out , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[60]  Gustavo Augusto,et al.  A MULTI-OBJECTIVE APPROACH FOR THE REGRESSION TEST CASE SELECTION PROBLEM , 2009 .

[61]  Yuanyuan Zhang,et al.  Search-based software engineering: Trends, techniques and applications , 2012, CSUR.

[62]  Francisco Luna,et al.  jMetal: a Java Framework for Developing Multi-Objective Optimization Metaheuristics , 2006 .

[63]  Jim Nilsson,et al.  An in-depth look at computer performance growth , 2005, CARN.

[64]  Mary Lou Soffa,et al.  TimeAware test suite prioritization , 2006, ISSTA '06.

[65]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.