The executable experimental template pattern for the systematic comparison of metaheuristics: Extended Abstract

Geoffrey Neumann Computing Science and Mathematics University of Stirling FK9 4LA Scotland UK gkn@cs.stir.ac.uk Jerry Swan Computing Science and Mathematics University of Stirling FK9 4LA Scotland UK jsw@cs.stir.ac.uk Mark Harman Department of Computer Science University College London, London, WC1E 6BT, UK. mark.harman@ucl.ac.uk John A. Clark Department of Computer Science and YYCSA University of York York, YO10 5GH, UK. john.clark@cs.york.ac.uk

[1]  E. Brunner,et al.  The Nonparametric Behrens‐Fisher Problem: Asymptotic Theory and a Small‐Sample Approximation , 2000 .

[2]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[3]  H. Levene.  Robust tests for equality of variances , 1961 .

[4]  Mark Harman,et al.  Searching for better configurations: a rigorous approach to clone evaluation , 2013, ESEC/FSE 2013.

[5]  Lionel C. Briand,et al.  A practical guide for using statistical tests to assess randomized algorithms in software engineering , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[6]  Peter Dalgaard,et al.  R Development Core Team (2010): R: A language and environment for statistical computing , 2010 .

[7]  John N. Hooker,et al.  Needed: An Empirical Science of Algorithms , 1994, Oper. Res..

[8]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[9]  S. Shapiro,et al.  A Comparative Study of Various Tests for Normality , 1968 .

[10]  S. Shapiro,et al.  An Approximate Analysis of Variance Test for Normality , 1972 .

[11]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[12]  Lionel C. Briand,et al.  A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering , 2014, Softw. Test. Verification Reliab..

[13]  Graham Kendall,et al.  A Classification of Hyper-heuristic Approaches , 2010 .

[14]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[15]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[16]  Mark Harman,et al.  Search Based Software Engineering: Techniques, Taxonomy, Tutorial , 2010, LASER Summer School.

[17]  E. S. Pearson,et al.  ON THE USE AND INTERPRETATION OF CERTAIN TEST CRITERIA FOR PURPOSES OF STATISTICAL INFERENCE PART I , 1928 .