The Human Competitiveness of Search Based Software Engineering

This paper reports a comprehensive experimental study regarding the human competitiveness of search based software engineering (SBSE). The experiments were performed over four well-known SBSE problem formulations: next release problem, multi-objective next release problem, workgroup formation problem and the multi-objective test case selection problem. For each of these problems, two instances, with increasing sizes, were synthetically generated and solved by both metaheuristics and human subjects. A total of 63 professional software engineers participated in the experiment by solving some or all problem instances, producing together 128 responses. The comparison analysis strongly suggests that the results generated by search based software engineering can be said to be human competitive.

[1]  Greg Burdett,et al.  A quantitative approach to the formation of workgroups , 1995, SIGCPR '95.

[2]  Francisco Fernández de Vega,et al.  Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software , 2009, Inf. Softw. Technol..

[3]  Yuanyuan Zhang,et al.  “Fairness Analysis” in Requirements Assignments , 2008, 2008 16th IEEE International Requirements Engineering Conference.

[4]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[5]  Mark Harman,et al.  Pareto efficient multi-objective test case selection , 2007, ISSTA '07.

[6]  Giuliano Antoniol,et al.  Software project planning for robustness and completion time in the presence of uncertainty using multi objective search based software engineering , 2009, GECCO.

[7]  Jerffeson Teixeira de Souza,et al.  A New Approach to the Software Release Planning , 2009, 2009 XXIII Brazilian Symposium on Software Engineering.

[8]  Enrique Alba,et al.  A Study of the Multi-objective Next Release Problem , 2009, 2009 1st International Symposium on Search Based Software Engineering.

[9]  Yuanyuan Zhang,et al.  Search Based Software Engineering: A Comprehensive Analysis and Review of Trends Techniques and Applications , 2009 .

[10]  Yuanyuan Zhang,et al.  The multi-objective next release problem , 2007, GECCO '07.

[11]  Yuanyuan Zhang,et al.  A search based approach to fairness analysis in requirement assignments to aid negotiation, mediation and decision making , 2009, Requirements Engineering.

[12]  Jim Mc Elroy,et al.  Software release planning with time-dependent value functions and flexible release dates , 2007, ICSE 2007.

[13]  John R. Koza.  Genetic Programming III - Darwinian Invention and Problem Solving , 1999, Evolutionary Computation.

[14]  Yuanyuan Zhang,et al.  Search Based Requirements Optimisation: Existing Work and Challenges , 2008, REFSQ.

[15]  Mark Harman,et al.  Search Based Approaches to Component Selection and Prioritization for the Next Release Problem , 2006, 2006 22nd IEEE International Conference on Software Maintenance.

[16]  Tao Xie,et al.  Time-aware test-case prioritization using integer linear programming , 2009, ISSTA.

[17]  Márcio de Oliveira Barros,et al.  Staffing a software project: a constraint satisfaction approach , 2005, ACM SIGSOFT Softw. Eng. Notes.

[18]  Enrique Alba,et al.  A Study of Convergence Speed in Multi-objective Metaheuristics , 2008, PPSN.

[19]  Mark Harman,et al.  Using hybrid algorithm for Pareto efficient multi-objective test suite minimisation , 2010, J. Syst. Softw..

[20]  Shin Yoo,et al.  Search based data sensitivity analysis applied to requirement engineering , 2009, GECCO.

[21]  Mark Harman,et al.  The Current State and Future of Search Based Software Engineering , 2007, Future of Software Engineering (FOSE '07).

[22]  Phil McMinn,et al.  Search-based failure discovery using testability transformations to generate pseudo-oracles , 2009, GECCO.

[23]  Chiou Peng Lam,et al.  Evolving the Quality of a Model Based Test Suite , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[24]  Mark Harman,et al.  Measuring and Improving Latency to Avoid Test Suite Wear Out , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[25]  Mel Ó Cinnéide,et al.  Journal of Software Maintenance and Evolution: Research and Practice Search-based Refactoring: an Empirical Study , 2022 .

[26]  Mel Ó Cinnéide,et al.  Search-based refactoring: an empirical study , 2008 .

[27]  Thesis.  Sensitivity Analysis in Multi-objective Next Release Problem and Fairness Analysis in Software Requirements Engineering , 2007 .

[28]  Mark Harman,et al.  Regression Testing Minimisation, Selection and Prioritisation - A Survey , 2009 .

[29]  John A. Clark,et al.  Efficient Software Verification: Statistical Testing Using Automated Search , 2010, IEEE Transactions on Software Engineering.

[30]  Janice Singer,et al.  Ethical Issues in Empirical Studies of Software Engineering , 2002, IEEE Trans. Software Eng..

[31]  Lionel C. Briand,et al.  Solving the Class Responsibility Assignment Problem in Object-Oriented Analysis with Multi-Objective Genetic Algorithms , 2010, IEEE Transactions on Software Engineering.

[32]  Mark Harman,et al.  Search Based Software Engineering for Program Comprehension , 2007, 15th IEEE International Conference on Program Comprehension (ICPC '07).

[33]  Kalyanmoy Deb,et al.  A fast and elitist multiobjective genetic algorithm: NSGA-II , 2002, IEEE Trans. Evol. Comput..

[34]  Huai Wang.  On the Construction of Context-Aware Test Suites † , 2010 .

[35]  Mark Harman.  Automated Test Data Generation using Search Based Software Engineering , 2007, Second International Workshop on Automation of Software Test (AST '07).

[36]  Victor J. Rayward-Smith,et al.  The next release problem , 2001, Inf. Softw. Technol..

[37]  Ronald E. Anderson.  ACM code of ethics and professional conduct , 1992, CACM.

[38]  Günther Ruhe,et al.  Software release planning for evolving systems , 2005, Innovations in Systems and Software Engineering.

[39]  Gustavo Augusto Lima de Campos,et al.  Automated Test Case Prioritization with Reactive GRASP , 2010, Adv. Softw. Eng..

[40]  Lu Zhang,et al.  An experimental study of four typical test suite reduction techniques , 2008, Inf. Softw. Technol..

[41]  Enrique Alba,et al.  MOCell: A cellular genetic algorithm for multiobjective optimization , 2009, Int. J. Intell. Syst..

[42]  Günther Ruhe,et al.  A systematic approach for solving the wicked problem of software release planning , 2007, Soft Comput..

[43]  Huai Wang,et al.  Weaving Context Sensitivity into Test Suite Construction , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[44]  G. Uma,et al.  Knowledge Mining of Test Case System , 2010 .

[45]  Hrushikesha Mohanty,et al.  Automated Test Scenario Selection Based on Levenshtein Distance , 2010, ICDCIT.

[46]  Mark Harman,et al.  Multi Objective Higher Order Mutation Testing with Genetic Programming , 2009, 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques.

[47]  Swarnendu Biswas,et al.  A model-based regression test selection approach for embedded applications , 2009, SOEN.

[48]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[49]  Mark Harman,et al.  Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[50]  Xin Yao,et al.  Software Module Clustering as a Multi-Objective Search Problem , 2011, IEEE Transactions on Software Engineering.