Search-Based Software Engineering: 12th International Symposium, SSBSE 2020, Bari, Italy, October 7–8, 2020, Proceedings

“Testing can be quite effective for showing the presence of bugs, but is hopelessly inadequate for showing their absence”. This famous remark, which was made by Dijkstra, has often been used to indicate a dichotomy between testing and verification. From a practitioner’s point of view, however, there is not much difference in the ways testing and verification techniques may be used in practice. While engineers would try to demonstrate that their systems are correct (verification), they often find themselves in a situation where they have to prioritize bug finding (testing). As a result, the choice to go for formal verification versus testing is largely driven by the practical needs and the context specificities. In this keynote, I will focus on search-based software testing (SBST) and review some recent research that combines ideas from the SBST and the formal verification communities to improve the analysis of models of cyber physical systems (CPS). I will present an empirical study that compares CPS model testing and verification, a search-based testing approach for compute-intensive CPS models that builds on a wellknown formal verification framework, and a technique to automatically generate formal environment assumptions for CPS models using search algorithms and genetic programming.

[1]  Bogdan Filipic,et al.  DEMO: Differential Evolution for Multiobjective Optimization , 2005, EMO.

[2]  James M. Bieman,et al.  An Assessment of the Quality of Automated Program Operator Repair , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[3]  Harald C. Gall,et al.  Exploring the integration of user feedback in automated testing of Android applications , 2018, 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER).

[4]  Alessandra Gorla,et al.  Checking app behavior against app descriptions , 2014, ICSE.

[5]  Norbert Siegmund,et al.  Transfer learning for performance modeling of configurable systems: An exploratory analysis , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[6]  Myra B. Cohen,et al.  Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback DrivenAdaptive Approach , 2014, IEEE Transactions on Software Engineering.

[7]  John Joseph Chilenski,et al.  An Investigation of Three Forms of the Modified Condition Decision Coverage (MCDC) Criterion , 2001 .

[8]  Yuanyuan Zhang,et al.  Inferring Test Models from Kate's Bug Reports Using Multi-objective Search , 2015, SSBSE.

[9]  Markus Herrmannsdoerfer,et al.  GMF: A Model Migration Case for the Transformation Tool Contest , 2011, TTC.

[10]  Alain Pétrowski,et al.  A clearing procedure as a niching method for genetic algorithms , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[11]  N. Hopper,et al.  AppGP: an alternative structural representation for GP , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[12]  Gordon Fraser,et al.  Handling test length bloat , 2013, Softw. Test. Verification Reliab..

[13]  Edgar Galván López,et al.  locoGP: Improving Performance by Genetic Programming Java Source Code , 2015, GECCO.

[14]  Bogdan Korel,et al.  Automated Software Test Data Generation , 1990, IEEE Trans. Software Eng..

[15]  David E. Goldberg,et al.  Genetic Algorithms, Tournament Selection, and the Effects of Noise , 1995, Complex Syst..

[16]  Claire Pagetti,et al.  Off-line mapping of multi-rate dependent task sets to many-core platforms , 2015, Real-Time Systems.

[17]  B. Gruschko Towards Synchronizing Models with Evolving Metamodels , 2007 .

[18]  Sam Malek,et al.  EvoDroid: segmented evolutionary testing of Android apps , 2014, SIGSOFT FSE.

[19]  P. Rousseeuw Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[20]  Xavier Devroey,et al.  Replication package of Basic Block Coverage for Search-Based Crash Reproduction , 2020 .

[21]  Diana Franklin,et al.  Hairball: lint-inspired static analysis of scratch projects , 2013, SIGCSE '13.

[22]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[23]  Conor Ryan,et al.  Seeding Grammars in Grammatical Evolution to Improve Search Based Software Testing , 2020, EuroGP.

[24]  John P. Lehoczky,et al.  The rate monotonic scheduling algorithm: exact characterization and average case behavior , 1989, [1989] Proceedings. Real-Time Systems Symposium.

[25]  Federico Ciccozzi,et al.  On the concurrent versioning of metamodels and models: challenges and possible solutions , 2011, IWMCP '11.

[26]  Dirk Sudholt,et al.  On the benefits and risks of using fitness sharing for multimodal optimisation , 2019, Theor. Comput. Sci..

[27]  Adam Kiezun,et al.  Grammar-based whitebox fuzzing , 2008, PLDI '08.

[28]  Gregory Gay,et al.  The Effect of Program and Model Structure on the Effectiveness of MC/DC Test Adequacy Coverage , 2016, ACM Trans. Softw. Eng. Methodol..

[29]  Fabrice Kordon,et al.  polyDD: Towards a Framework Generalizing Decision Diagrams , 2010, 2010 10th International Conference on Application of Concurrency to System Design.

[30]  Mark Harman,et al.  Approximate Oracles and Synergy in Software Energy Search Spaces , 2019, IEEE Transactions on Software Engineering.

[31]  Gregory Gay,et al.  Using Search-Based Test Generation to Discover Real Faults in Guava , 2017, SSBSE.

[32]  Yuanyuan Zhang,et al.  Feature lifecycles as they spread, migrate, remain, and die in App Stores , 2015, 2015 IEEE 23rd International Requirements Engineering Conference (RE).

[33]  Houari A. Sahraoui,et al.  Automated metamodel/model co-evolution: A search-based approach , 2019, Inf. Softw. Technol..

[34]  Don S. Batory,et al.  Finding near-optimal configurations in product lines by random sampling , 2017, ESEC/SIGSOFT FSE.

[35]  Paolo Tonella,et al.  Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets , 2018, IEEE Transactions on Software Engineering.

[36]  Gian Mauricio Fritsche,et al.  A Hyper-Heuristic for the Multi-Objective Integration and Test Order Problem , 2015, GECCO.

[37]  Xiangyu Zhang,et al.  Automatic Text Input Generation for Mobile Testing , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[38]  Yuanyuan Zhang,et al.  A Survey of App Store Analysis for Software Engineering , 2017, IEEE Transactions on Software Engineering.

[39]  Matias Martinez,et al.  Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset , 2016, Empirical Software Engineering.

[40]  Shin Yoo,et al.  Comparing line and AST granularity level for program repair using PyGGI , 2018, GI@ICSE.

[41]  Andreas Zeller,et al.  Why Programs Fail, Second Edition: A Guide to Systematic Debugging , 2009 .

[42]  Márcio de Oliveira Barros,et al.  What Can a Big Program Teach Us about Optimization? , 2013, SSBSE.

[43]  Bruno Sareni,et al.  Fitness sharing and niching methods revisited , 1998, IEEE Trans. Evol. Comput..

[44]  Yuriy Brun,et al.  Behavioral Execution Comparison: Are Tests Representative of Field Behavior? , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[45]  Kalyanmoy Deb,et al.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II , 2000, PPSN.

[46]  Xavier Devroey,et al.  An Application of Model Seeding to Search-Based Unit Test Generation for Gson , 2020, SSBSE.

[47]  Gordon Fraser,et al.  Seeding strategies in search‐based unit test generation , 2016, Softw. Test. Verification Reliab..

[48]  Abdelwahab Hamou-Lhadj,et al.  JCHARMING: A bug reproduction approach using crash traces and directed model checking , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[49]  Arie van Deursen,et al.  Search‐based crash reproduction using behavioural model seeding , 2019, Softw. Test. Verification Reliab..

[50]  René Just,et al.  Unit Testing Tool Competition — Round Four , 2016, 2016 IEEE/ACM 9th International Workshop on Search-Based Software Testing (SBST).

[51]  Bart Meyers,et al.  A framework for evolution of modelling languages , 2011, Sci. Comput. Program..

[52]  Gordon Fraser,et al.  A detailed investigation of the effectiveness of whole test suite generation , 2017, Empirical Software Engineering.

[53]  Andreas Zeller,et al.  Inputs from Hell: Generating Uncommon Inputs from Common Samples , 2018, ArXiv.

[54]  Paolo Tonella,et al.  Evolutionary testing of classes , 2004, ISSTA '04.

[55]  Manuel Wimmer,et al.  CARE - A Constraint-Based Approach for Re-Establishing Conformance-Relationships , 2014, APCCM.

[56]  Gregory Gay,et al.  Observable modified condition/decision coverage , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[57]  Ujjwal Maulik,et al.  Genetic algorithm-based clustering technique , 2000, Pattern Recognit..

[58]  Gordon Fraser,et al.  Testing scratch programs automatically , 2019, ESEC/SIGSOFT FSE.

[59]  Sven Apel,et al.  Using bad learners to find good configurations , 2017, ESEC/SIGSOFT FSE.

[60]  Arie van Deursen,et al.  Monitoring-aware IDEs , 2019, ESEC/SIGSOFT FSE.

[61]  Rishabh Singh,et al.  Learn&Fuzz: Machine learning for input fuzzing , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[62]  Thomas Vogel,et al.  Does Diversity Improve the Test Suite Generation for Mobile Applications? , 2019, SSBSE.

[63]  Claire Le Goues,et al.  GenProg: A Generic Method for Automatic Software Repair , 2012, IEEE Transactions on Software Engineering.

[64]  Yves Le Traon,et al.  An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation That Avoids the Unreliable Clean Program Assumption , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[65]  Tim Menzies,et al.  Bellwethers: A Baseline Method for Transfer Learning , 2017, IEEE Transactions on Software Engineering.

[66]  Bernhard Schätz,et al.  Graph and model transformation tools for model migration , 2014, Software & Systems Modeling.

[67]  Eli Tilevich,et al.  Understanding recurring quality problems and their impact on code sharing in block-based software , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[68]  Paolo Tonella,et al.  Symbolic search-based testing , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[69]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[70]  Gregory Gay,et al.  The Risks of Coverage-Directed Test Case Generation , 2015, IEEE Transactions on Software Engineering.

[71]  Kristen Walcott-Justice,et al.  Empirically Evaluating the Quality of Automatically Generated and Manually Written Test Suites , 2014, 2014 14th International Conference on Quality Software.

[72]  Angelo Gargantini,et al.  AURORA: AUtomatic RObustness coveRage Analysis Tool , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[73]  Markus Wagner,et al.  Data-Driven Search-Based Software Engineering , 2018, 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR).

[74]  Tao Xie,et al.  Improving Structural Testing of Object-Oriented Programs via Integrating Evolutionary Testing and Symbolic Execution , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[75]  T. Stützle,et al.  Iterated Local Search: Framework and Applications , 2018, Handbook of Metaheuristics.

[76]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[77]  Patrice Godefroid,et al.  SAGE: Whitebox Fuzzing for Security Testing , 2012, ACM Queue.

[78]  Koushik Sen,et al.  Saffron: Adaptive Grammar-based Fuzzing for Worst-Case Analysis , 2019, SOEN.

[79]  Gordon Fraser,et al.  EvoSuite: automatic test suite generation for object-oriented software , 2011, ESEC/FSE '11.

[80]  Anders Rune Jensen,et al.  Optimal Task Graph Scheduling with Binary Decision Diagrams , 2004 .

[81]  Yuriy Brun,et al.  Quality of Automated Program Repair on Real-World Defects , 2022, IEEE Transactions on Software Engineering.

[82]  Chris Cummins,et al.  Compiler fuzzing through deep learning , 2018, ISSTA.

[83]  Christian Wolff,et al.  Similarity-based analyses on software applications: A systematic literature review , 2020, J. Syst. Softw..

[84]  Andrew Ruef,et al.  Evaluating Fuzz Testing , 2018, CCS.

[85]  Luca Scrucca,et al.  GA: A Package for Genetic Algorithms in R , 2013 .

[86]  Dave Towey,et al.  Forgetting Test Cases , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[87]  Gordon Fraser,et al.  Combining Multiple Coverage Criteria in Search-Based Unit Test Generation , 2015, SSBSE.

[88]  Zibin Zheng,et al.  Configuring Software Product Lines by Combining Many-Objective Optimization and SAT Solvers , 2018, ACM Trans. Softw. Eng. Methodol..

[89]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[90]  A. Vargha,et al.  A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong , 2000 .

[91]  A Pollatsek,et al.  On the use of counterbalanced designs in cognitive research: a suggestion for a better and more powerful analysis. , 1995, Journal of experimental psychology. Learning, memory, and cognition.

[92]  Xin Yao,et al.  Software Module Clustering as a Multi-Objective Search Problem , 2011, IEEE Transactions on Software Engineering.

[93]  Wael Kessentini,et al.  Automated Metamodel/Model Co-Evolution using a Multi-Objective Optimization Approach , 2016, SRC@MoDELS.

[94]  André Augusto Ciré,et al.  Multivalued Decision Diagrams for Sequencing Problems , 2013, Oper. Res..

[95]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[96]  Arie van Deursen,et al.  Towards Integration-Level Test Case Generation Using Call Site Information , 2020, ArXiv.

[97]  Mordechai Ben-Ari,et al.  Habits of programming in scratch , 2011, ITiCSE '11.

[98]  Ali Shahbazi,et al.  Diversity-Based Automated Test Case Generation , 2015 .

[99]  Fabrice Kordon,et al.  Computing a Hierarchical Static Order for Decision Diagram-Based Representation from P/T Nets , 2012, Trans. Petri Nets Other Model. Concurr..

[100]  Ge Yu,et al.  Exact Schedulability Analysis for Static-Priority Global Multiprocessor Scheduling Using Model-Checking , 2007, SEUS.

[101]  J Hayhurst Kelly,et al.  A Practical Tutorial on Modified Condition/Decision Coverage , 2001 .

[102]  Gregory Gay,et al.  Choosing the fitness function for the job: Automated generation of test suites that detect real faults , 2019, Softw. Test. Verification Reliab..

[103]  Jinyi Guo,et al.  Elitism and Distance Strategy for Selection of Evolutionary Algorithms , 2018, IEEE Access.

[104]  Aurora Ramírez,et al.  A Systematic Review of Interaction in Search-Based Software Engineering , 2019, IEEE Transactions on Software Engineering.

[105]  Andrea De Lucia,et al.  Automatic test case generation: what if test code quality matters? , 2016, ISSTA.

[106]  Felienne Hermans,et al.  How Kids Code and How We Know: An Exploratory Study on the Scratch Repository , 2016, ICER.

[107]  Murat Peker A fully customizable hardware implementation for general purpose genetic algorithms , 2018, Appl. Soft Comput..

[108]  Alex Alves Freitas,et al.  A Survey of Evolutionary Algorithms for Clustering , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[109]  Phil McMinn,et al.  Search‐based software test data generation: a survey , 2004, Softw. Test. Verification Reliab..

[110]  Mark Harman,et al.  Babel Pidgin: SBSE Can Grow and Graft Entirely New Functionality into a Real World System , 2014, SSBSE.

[111]  Alain Denise,et al.  A new way of automating statistical testing methods , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[112]  Eli Tilevich,et al.  Code Quality Improvement for All: Automated Refactoring for Scratch , 2019, 2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[113]  Gordon Fraser,et al.  Do Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[114]  Alessandro Orso,et al.  Mimicking User Behavior to Improve In-House Test Suites , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).

[115]  Roxana Geambasu,et al.  Pythia: Grammar-Based Fuzzing of REST APIs with Coverage-guided Feedback and Learning-based Mutations , 2020, ArXiv.

[116]  Radu Calinescu,et al.  Dynamic QoS Management and Optimization in Service-Based Systems , 2011, IEEE Transactions on Software Engineering.

[117]  David Clark,et al.  Test Set Diameter: Quantifying the Diversity of Sets of Test Cases , 2015, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[118]  Dirk Sudholt,et al.  On the Runtime Analysis of the Clearing Diversity-Preserving Mechanism , 2018, Evolutionary Computation.

[119]  Mohsen Guizani,et al.  Scratch Analysis Tool(SAT): A Modern Scratch Project Analysis Tool based on ANTLR to Assess Computational Thinking Skills , 2018, 2018 14th International Wireless Communications & Mobile Computing Conference (IWCMC).

[120]  Foutse Khomh,et al.  Efficient refactoring scheduling based on partial order reduction , 2018, J. Syst. Softw..

[121]  Gordon Fraser,et al.  Improving search-based test suite generation with dynamic symbolic execution , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[122]  Gregory Gay Challenges in Using Search-Based Test Generation to Identify Real Faults in Mockito , 2016, SSBSE.

[123]  GORDON FRASER,et al.  A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite , 2014, ACM Trans. Softw. Eng. Methodol..

[124]  Houari A. Sahraoui,et al.  A Cooperative Parallel Search-Based Software Engineering Approach for Code-Smells Detection , 2014, IEEE Transactions on Software Engineering.

[125]  E. Myers,et al.  Basic local alignment search tool. , 1990, Journal of molecular biology.

[126]  Fan Wu,et al.  Deep Parameter Optimisation , 2015, GECCO.

[127]  Christoph Frädrich,et al.  Common Bugs in Scratch Programs , 2020, ITiCSE.

[128]  Annibale Panichella,et al.  A Systematic Comparison of Search Algorithms for Topic Modelling - A Study on Duplicate Bug Report Identification , 2019, SSBSE.

[129]  Gregorio Robles,et al.  Dr. Scratch: a Web Tool to Automatically Evaluate Scratch Projects , 2015, WiPSCE.

[130]  Jason Mars,et al.  Scenario Based Optimization: A Framework for Statically Enabling Online Optimizations , 2009, 2009 International Symposium on Code Generation and Optimization.

[131]  Gordon Fraser,et al.  1600 faults in 100 projects: automatically finding faults while achieving high coverage with EvoSuite , 2015, Empirical Software Engineering.

[132]  Kaisa Miettinen,et al.  On initial populations of a genetic algorithm for continuous optimization problems , 2007, J. Glob. Optim..

[133]  Dirk Sudholt,et al.  Design and analysis of different alternating variable searches for search-based software testing , 2015, Theor. Comput. Sci..

[134]  Alexander Mendiburu,et al.  Multi-start Methods , 2018, Handbook of Heuristics.

[135]  Andreas Zeller,et al.  Reconstructing Core Dumps , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[136]  Myra B. Cohen,et al.  The Maturation of Search-Based Software Testing: Successes and Challenges , 2019, 2019 IEEE/ACM 12th International Workshop on Search-Based Software Testing (SBST).

[137]  Fan Wu,et al.  Darwinian data structure selection , 2017, ESEC/SIGSOFT FSE.

[138]  Mark Harman,et al.  Improving CUDA DNA Analysis Software with Genetic Programming , 2015, GECCO.

[139]  Doo-Hwan Bae,et al.  An efficient approach to identify multiple and independent Move Method refactoring candidates , 2015, Inf. Softw. Technol..

[140]  Mark Harman,et al.  FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution , 2010, ICTSS.

[141]  Gabor Karsai,et al.  Automatic Domain Model Migration to Manage Metamodel Evolution , 2009, MoDELS.

[142]  Arie van Deursen,et al.  Single-objective Versus Multi-objectivized Optimization for Evolutionary Crash Reproduction , 2018, SSBSE.

[143]  Xuejun Yang,et al.  Finding and understanding bugs in C compilers , 2011, PLDI '11.

[144]  Paolo Falcarin,et al.  Search Based Clustering for Protecting Software with Diversified Updates , 2016, SSBSE.

[145]  A. E. Eiben,et al.  On-Line, On-Board Evolution of Robot Controllers , 2009, Artificial Evolution.

[146]  Antonio Cicchetti,et al.  A generic in-place transformation-based approach to structured model co-evolution , 2011, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..

[147]  Oliver Krauss,et al.  Evolving sqrt into 1/x via software data maintenance , 2020, GECCO Companion.

[148]  André Augusto Ciré,et al.  Decision diagrams for optimization , 2018, Constraints.

[149]  Patrice Godefroid,et al.  Fuzzing: hack, art, and science , 2020, Commun. ACM.

[150]  Alex Groce,et al.  Coverage and Its Discontents , 2014, Onward!.

[151]  Andreas Zeller,et al.  Search-based system testing: high coverage, no false alarms , 2012, ISSTA 2012.

[152]  Zhenkai Liang,et al.  BitBlaze: A New Approach to Computer Security via Binary Analysis , 2008, ICISS.

[153]  Jian Song,et al.  An Automated Approach to Estimating Code Coverage Measures via Execution Logs , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[154]  Myra B. Cohen,et al.  Navigating the Maze: The Impact of Configurability in Bioinformatics Software , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[155]  K. V. Hanford,et al.  Automatic Generation of Test Cases , 1970, IBM Syst. J..

[156]  Yuanyuan Zhang,et al.  Mobile App and App Store Analysis, Testing, and Optimisation , 2016, 2016 IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft).

[157]  Marjan Mernik,et al.  Exploration and exploitation in evolutionary algorithms: A survey , 2013, CSUR.

[158]  Sven Apel,et al.  Performance-influence models for highly configurable systems , 2015, ESEC/SIGSOFT FSE.

[159]  Pierre-Yves Schobbens,et al.  Statistical prioritization for software product line testing: an experience report , 2015, Software & Systems Modeling.

[160]  C. Spearman The proof and measurement of association between two things. , 2015, International journal of epidemiology.

[161]  Zita Vale,et al.  A platform for testing the performance of metaheuristics solving the energy resource management problem in smart grids , 2018 .

[162]  Gregory Gay,et al.  Ensuring the Observability of Structural Test Obligations , 2020, IEEE Transactions on Software Engineering.

[163]  Harald C. Gall,et al.  The impact of test case summaries on bug fixing performance: an empirical investigation , 2016, PeerJ Prepr..

[164]  Gabor Karsai,et al.  A domain-specific visual language for domain model evolution , 2004, J. Vis. Lang. Comput..

[165]  Lori L. Pollock,et al.  Configuring effective navigation models and abstract test cases for web applications by analysing user behaviour , 2013, Softw. Test. Verification Reliab..

[166]  G LarsenKim,et al.  Optimal scheduling using priced timed automata , 2005 .

[167]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[168]  Myra B. Cohen,et al.  Configuration-aware regression testing: an empirical study of sampling and prioritization , 2008, ISSTA '08.

[169]  Giovanni Vigna,et al.  Detecting malicious JavaScript code in Mozilla , 2005, 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'05).

[170]  Gunter Saake,et al.  On essential configuration complexity: Measuring interactions in highly-configurable systems , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[171]  Conor Ryan,et al.  Grammatical evolution , 2007, GECCO '07.

[172]  Yves Le Traon,et al.  Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[173]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[174]  Andrea Arcuri,et al.  RESTful API Automated Test Case Generation with EvoMaster , 2019, ACM Trans. Softw. Eng. Methodol..

[175]  Peter J. Fleming,et al.  Diversity Management in Evolutionary Many-Objective Optimization , 2011, IEEE Transactions on Evolutionary Computation.

[176]  Andrew E. Santosa,et al.  Smart Greybox Fuzzing , 2018, IEEE Transactions on Software Engineering.

[177]  Kim G. Larsen,et al.  Schedulability of Herschel revisited using statistical model checking , 2014, International Journal on Software Tools for Technology Transfer.

[178]  Ning Chen,et al.  STAR: Stack Trace Based Automatic Crash Reproduction via Symbolic Execution , 2015, IEEE Transactions on Software Engineering.

[179]  Viet Anh Nguyen,et al.  Cache-Conscious Offline Real-Time Task Scheduling for Multi-Core Processors , 2017, ECRTS.

[180]  Abhik Roychoudhury,et al.  Directed Greybox Fuzzing , 2017, CCS.

[181]  Tim Menzies,et al.  Too much automation? The bellwether effect and its implications for transfer learning , 2016, 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE).

[182]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[183]  Edmund M. Clarke,et al.  Counterexample-guided abstraction refinement , 2003, 10th International Symposium on Temporal Representation and Reasoning, 2003 and Fourth International Conference on Temporal Logic. Proceedings..

[184]  Jeff Tian,et al.  Measuring and Modeling Usage and Reliability for Statistical Web Testing , 2001, IEEE Trans. Software Eng..

[185]  Gordon Fraser,et al.  EvoSuite: On the Challenges of Test Case Generation in the Real World , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[186]  Gordon Fraser,et al.  Evolutionary Generation of Whole Test Suites , 2011, 2011 11th International Conference on Quality Software.

[187]  Thomas W. Price,et al.  iSnap: Towards Intelligent Tutoring in Novice Programming Environments , 2017, SIGCSE.

[188]  Giovanni Squillero,et al.  Divergence of character and premature convergence: A survey of methodologies for promoting diversity in evolutionary optimization , 2016, Inf. Sci..

[189]  Jacques Klein,et al.  Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines , 2014, IEEE Transactions on Software Engineering.

[190]  Martin Monperrus,et al.  Crash reproduction via test case mutation: let existing test cases help , 2015, ESEC/SIGSOFT FSE.

[191]  Jan Madsen,et al.  ARTS: a system-level framework for modeling MPSoC components and analysis of their causality , 2005, 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems.

[192]  Fearghal Morgan,et al.  Maintaining Healthy Population Diversity Using Adaptive Crossover, Mutation, and Selection , 2011, IEEE Transactions on Evolutionary Computation.

[193]  Michael D. Ernst,et al.  Evaluating and Improving Fault Localization , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[194]  Luca Maria Gambardella,et al.  A survey on metaheuristics for stochastic combinatorial optimization , 2009, Natural Computing.

[195]  Eli Tilevich,et al.  Quality Hound — An online code smell analyzer for scratch programs , 2017, 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[196]  Mark Harman,et al.  Genetic Improvement of Software: A Comprehensive Survey , 2018, IEEE Transactions on Evolutionary Computation.

[197]  Gordon Fraser,et al.  Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation? , 2015, GECCO.

[198]  Gordon Fraser,et al.  Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study , 2015, ACM Trans. Softw. Eng. Methodol..

[199]  Markus Wagner,et al.  A Generic Bet-and-Run Strategy for Speeding Up Stochastic Local Search , 2017, AAAI.

[200]  Frédéric Boniol,et al.  Off-line (Optimal) multiprocessor scheduling of dependent periodic tasks , 2012, SAC '12.

[201]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[202]  Xin Yao,et al.  A multi-agent evolutionary algorithm for software module clustering problems , 2016, Soft Computing.

[203]  Houari A. Sahraoui,et al.  Integrating the Designer in-the-loop for Metamodel/Model Co-Evolution via Interactive Computational Search , 2018, MoDELS.

[204]  Andreas Zeller,et al.  Exploiting Common Object Usage in Test Case Generation , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[205]  Myra B. Cohen,et al.  Evaluating improvements to a meta-heuristic search for constrained interaction testing , 2011, Empirical Software Engineering.

[206]  D. Bertsimas,et al.  Robust and Data-Driven Optimization: Modern Decision-Making Under Uncertainty , 2006 .

[207]  Maleknaz Nayebi,et al.  More insight from being more focused: analysis of clustered market apps , 2016, WAMA@SIGSOFT FSE.

[208]  Didier Buchs,et al.  Solving Schedulability as a Search Space Problem with Decision Diagrams , 2020, SSBSE.

[209]  David E. Johnson ITCH: Individual Testing of Computer Homework for Scratch Assignments , 2016, SIGCSE.

[210]  Herbert Bos,et al.  IFuzzer: An Evolutionary Interpreter Fuzzer Using Genetic Programming , 2016, ESORICS.

[211]  Ayse T. Daloglu,et al.  An improved genetic algorithm with initial population strategy and self-adaptive member grouping , 2008 .

[212]  Andreas Zeller,et al.  Mining Input Grammars with AUTOGRAM , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C).

[213]  Mark Harman,et al.  Automated software transplantation , 2015, ISSTA.

[214]  Barbara Korousic-Seljak Task scheduling policies for real-time systems , 1994, Microprocess. Microsystems.

[215]  Didier Buchs,et al.  Verifying Multi-core Schedulability with Data Decision Diagrams , 2016, SERENE.

[216]  A. Jefferson Offutt,et al.  Is bytecode instrumentation as good as source code instrumentation: An empirical study with industrial tools (Experience Report) , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[217]  Myra B. Cohen,et al.  Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[218]  Ofer M. Shir,et al.  Niching in Evolutionary Algorithms , 2012, Handbook of Natural Computing.

[219]  Mariano Ceccato,et al.  Do Automatically Generated Test Cases Make Debugging Easier? An Experimental Assessment of Debugging Effectiveness and Efficiency , 2015, ACM Trans. Softw. Eng. Methodol..

[220]  Sven Apel,et al.  A Comparison of 10 Sampling Algorithms for Configurable Systems , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[221]  Hadi Hemmati,et al.  Revisiting Hyper-Parameter Tuning for Search-based Test Data Generation , 2019, SSBSE.

[222]  Kathryn T. Stolee,et al.  Smells in block-based programming languages , 2016, 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC).

[223]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[224]  Paolo Tonella,et al.  Reformulating Branch Coverage as a Many-Objective Optimization Problem , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[225]  Jean Bézivin,et al.  Managing Model Adaptation by Precise Detection of Metamodel Changes , 2009, ECMDA-FA.

[226]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[227]  Nachol Chaiyaratana,et al.  Effects of diversity control in single-objective and multi-objective genetic algorithms , 2007, J. Heuristics.

[228]  I. Mackay,et al.  Evolution of dominance , 1979, Nature.

[229]  Mitra Nasri,et al.  An Exact Schedulability Test for Non-Preemptive Self-Suspending Real-Time Tasks , 2019, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[230]  Paolo Tonella,et al.  Statistical testing of Web applications , 2004, J. Softw. Maintenance Res. Pract..

[231]  Mozhan Soltani,et al.  A benchmark-based evaluation of search-based crash reproduction , 2019, Empirical Software Engineering.

[232]  Gordon Fraser,et al.  An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[233]  Andreas Zeller,et al.  Fuzzing with Code Fragments , 2012, USENIX Security Symposium.

[234]  John Maloney,et al.  The Scratch Programming Language and Environment , 2010, TOCE.

[235]  Foutse Khomh,et al.  On the use of developers' context for automatic refactoring of software anti-patterns , 2017, J. Syst. Softw..

[236]  Filomena Ferrucci,et al.  Using Hadoop MapReduce for Parallel Genetic Algorithms: A Comparison of the Global, Grid and Island Models , 2018, Evolutionary Computation.

[237]  Anthony Ventresque,et al.  Demo: PIT a Practical Mutation Testing Tool for Java , 2016 .

[238]  Ding Li,et al.  Integrated energy-directed test suite optimization , 2014, ISSTA 2014.

[239]  Justyna Petke,et al.  Software Improvement with Gin: A Case Study , 2019, SSBSE.

[240]  Shin Yoo,et al.  A new approach to distribute MOEA pareto front computation , 2020, GECCO Companion.

[241]  Licia Capra,et al.  Empirical comparison of text-based mobile apps similarity measurement techniques , 2019, Empirical Software Engineering.

[242]  Yuanyuan Zhang,et al.  Customer Rating Reactions Can Be Predicted Purely using App Features , 2018, 2018 IEEE 26th International Requirements Engineering Conference (RE).

[243]  Robert Feldt,et al.  Visualizing Test Diversity to Support Test Optimisation , 2018, 2018 25th Asia-Pacific Software Engineering Conference (APSEC).

[244]  Nasser M. Albunian,et al.  Diversity in Search-Based Unit Test Suite Generation , 2017, SSBSE.

[245]  Yuanyuan Zhang,et al.  Clustering Mobile Apps Based on Mined Textual Features , 2016, ESEM.

[246]  Georgios Fainekos,et al.  Falsification of Cyber-Physical Systems with Robustness Uncertainty Quantification Through Stochastic optimization with Adaptive Restart , 2019, 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE).

[247]  W. B. Langdon Big data driven genetic improvement for maintenance of legacy software systems , 2020 .

[248]  Federica Sarro Search-Based Predictive Modelling for Software Engineering: How Far Have We Gone? , 2019, SSBSE.

[249]  Myra B. Cohen,et al.  Failure Avoidance in Configurable Systems through Feature Locality , 2013, Assurances for Self-Adaptive Systems.

[250]  Emmanuelle Encrenaz-Tiphène,et al.  Data Decision Diagrams for Petri Net Analysis , 2002, ICATPN.

[251]  A. Jefferson Offutt,et al.  How strong is weak mutation? , 1991, TAV4.

[252]  Spiros Mancoridis,et al.  Automatic clustering of software systems using a genetic algorithm , 1999, STEP '99. Proceedings Ninth International Workshop Software Technology and Engineering Practice.

[253]  Yuriy Brun,et al.  Is the cure worse than the disease? overfitting in automated program repair , 2015, ESEC/SIGSOFT FSE.

[254]  Richard F. Paige,et al.  Model Migration with Epsilon Flock , 2010, ICMT@TOOLS.

[255]  Gordon Fraser,et al.  Achieving scalable mutation-based generation of whole test suites , 2015, Empirical Software Engineering.

[256]  Mark Harman,et al.  Using Genetic Improvement and Code Transplants to Specialise a C++ Program to a Problem Class , 2014, EuroGP.

[257]  Yue Jia,et al.  Sapienz: multi-objective automated testing for Android applications , 2016, ISSTA.

[258]  Frances E. Allen,et al.  Control-flow analysis , 2022 .

[259]  Matteo Fischetti,et al.  Exploiting Erraticism in Search , 2014, Oper. Res..

[260]  Henry Muccini,et al.  Self-Adaptation for Cyber-Physical Systems: A Systematic Literature Review , 2016, 2016 IEEE/ACM 11th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS).

[261]  Annibale Panichella,et al.  Search-Based Crash Reproduction and Its Impact on Debugging , 2020, IEEE Transactions on Software Engineering.

[262]  S. Ronald Duplicate genotypes in a genetic algorithm , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[263]  Jerffeson Souza,et al.  Code Naturalness to Assist Search Space Exploration in Search-Based Program Repair Methods , 2019, SSBSE.

[264]  Markus Wagner,et al.  Gin: genetic improvement research made easy , 2019, GECCO.

[265]  Paolo Tonella,et al.  A large scale empirical comparison of state-of-the-art search-based test case generators , 2018, Inf. Softw. Technol..

[266]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[267]  Markus Herrmannsdoerfer,et al.  Language Evolution in Practice: The History of GMF , 2009, SLE.

[268]  David Jackson,et al.  Promoting Phenotypic Diversity in Genetic Programming , 2010, PPSN.

[269]  Andy Zaidman,et al.  Unit test generation for common and uncommon behaviors: replication package , 2020 .

[270]  Armin Biere,et al.  Bounded Model Checking Using Satisfiability Solving , 2001, Formal Methods Syst. Des..

[271]  Gregory Gay,et al.  Learning How to Search: Generating Exception-Triggering Tests Through Adaptive Fitness Function Selection , 2020, 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST).

[272]  Myra B. Cohen,et al.  An orchestrated survey of methodologies for automated software test case generation , 2013, J. Syst. Softw..

[273]  David R. White GI in no time , 2017, GECCO.

[274]  Antonio Cicchetti,et al.  Automating Co-evolution in Model-Driven Engineering , 2008, 2008 12th International IEEE Enterprise Distributed Object Computing Conference.

[275]  Abhik Roychoudhury,et al.  Coverage-Based Greybox Fuzzing as Markov Chain , 2016, IEEE Transactions on Software Engineering.

[276]  Gordon Fraser,et al.  Test suite generation with memetic algorithms , 2013, GECCO '13.

[277]  Leonardo Vanneschi,et al.  Operator equalisation for bloat free genetic programming and a survey of bloat control methods , 2011, Genetic Programming and Evolvable Machines.

[278]  Suzette Person,et al.  A Flexible and Non-intrusive Approach for Computing Complex Structural Coverage Metrics , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[279]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[280]  Pedro A. Diaz-Gomez,et al.  Empirical Study: Initial Population Diversity and Genetic Algorithm Performance , 2007, Artificial Intelligence and Pattern Recognition.

[281]  Regina Hebig,et al.  Approaches to Co-Evolution of Metamodels and Models: A Survey , 2017, IEEE Transactions on Software Engineering.

[282]  Mark Harman,et al.  App Store Effects on Software Engineering Practices , 2019, IEEE Transactions on Software Engineering.

[283]  Barton P. Miller,et al.  An empirical study of the reliability of UNIX utilities , 1990, Commun. ACM.

[284]  Gordon Fraser,et al.  Modeling readability to improve unit tests , 2015, ESEC/SIGSOFT FSE.

[285]  Tim Menzies,et al.  Software Analytics: So What? , 2013, IEEE Softw..

[286]  A. Jefferson Offutt,et al.  Mutation 2000: uniting the orthogonal , 2001 .

[287]  Clifford J. Maloney,et al.  Systematic mistake analysis of digital computer programs , 1963, CACM.

[288]  Ludovico Iovino,et al.  On the Impact Significance of Metamodel Evolution in MDE , 2012, J. Object Technol..

[289]  Sergio Bittanti Model Identification and Data Analysis , 2019 .

[290]  Yang Liu,et al.  Superion: Grammar-Aware Greybox Fuzzing , 2018, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[291]  Corina S. Pasareanu,et al.  DifFuzz: Differential Fuzzing for Side-Channel Analysis , 2018, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[292]  Myra B. Cohen,et al.  An Orchestrated Survey on Automated Software Test Case Generation I , 2013 .

[293]  Mark Harman,et al.  Ieee Transactions on Evolutionary Computation 1 , 2022 .

[294]  Felienne Hermans,et al.  Do code smells hamper novice programming? A controlled experiment on Scratch programs , 2016, 2016 IEEE 24th International Conference on Program Comprehension (ICPC).

[295]  Kim G. Larsen,et al.  Optimal scheduling using priced timed automata , 2005, PERV.

[296]  Koushik Sen,et al.  FairFuzz: A Targeted Mutation Strategy for Increasing Greybox Fuzz Testing Coverage , 2018, 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[297]  Gordon Fraser,et al.  An empirical evaluation of evolutionary algorithms for unit test suite generation , 2018, Inf. Softw. Technol..

[298]  Michael D. Ernst,et al.  Comparing developer-provided to user-provided tests for fault localization and automated program repair , 2018, ISSTA.

[299]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[300]  Andrea Arcuri,et al.  It really does matter how you normalize the branch distance in search‐based software testing , 2013, Softw. Test. Verification Reliab..

[301]  Gordon Fraser,et al.  Does automated white-box test generation really help software testers? , 2013, ISSTA.

[302]  Ralf Lämmel,et al.  Automated Co-evolution of GMF Editor Models , 2010, SLE.

[303]  Christian Kästner,et al.  Learning to sample: exploiting similarities across environments to learn performance models for configurable systems , 2018, ESEC/SIGSOFT FSE.