A new perspective on the competent programmer hypothesis through the reproduction of bugs with repeated mutations

The competent programmer hypothesis states that most programmers are competent enough to create correct or almost correct source code. Because this implies that bugs should usually manifest through small variations of the correct code, the competent programmer hypothesis is one of the fundamental assumptions of mutation testing. Unfortunately, it is still unclear if the competent programmer hypothesis holds and past research presents contradictory claims. Within this article, we provide a new perspective on the competent programmer hypothesis and its relation to mutation testing. We try to re-create real-world bugs through chains of mutations to understand if there is a direct link between mutation testing and bugs. The lengths of these paths help us to understand if the source code is really almost correct, or if large variations are required. Our results indicate that while the competent programmer hypothesis seems to be true, mutation testing is missing important operators to generate representative real-world bugs. Copyright © 2021 John Wiley & Sons, Ltd.

[1]  Magnus C. Ohlsson,et al.  Experimentation in Software Engineering , 2000, The Kluwer International Series in Software Engineering.

[2]  A. Jefferson Offutt,et al.  Investigations of the software testing coupling effect , 1992, TSEM.

[3]  Akbar Siami Namin,et al.  The use of mutation in testing experiments and its sensitivity to external threats , 2011, ISSTA '11.

[4]  René Just,et al.  The major mutation framework: efficient and scalable mutation analysis for Java , 2014, ISSTA 2014.

[5]  A. Jefferson Offutt,et al.  Recognizing authors: an examination of the consistent programmer hypothesis , 2010, Softw. Test. Verification Reliab..

[6]  R. Lipton,et al.  Mutation analysis , 1998 .

[7]  Hans van Vliet,et al.  Proceedings of the the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering , 2009, FSE 2009.

[8]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[9]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[10]  A. Jefferson Offutt.  The Coupling Effect: Fact or Fiction , 1989, Symposium on Testing, Analysis, and Verification.

[11]  Andreas Zeller,et al.  It's not a bug, it's a feature: How misclassification impacts bug prediction , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[12]  Rina Dechter,et al.  Generalized best-first search strategies and the optimality of A* , 1985, JACM.

[13]  Andreas Zeller,et al.  The impact of tangled code changes on defect prediction models , 2015, Empirical Software Engineering.

[14]  MaYu-Seung,et al.  MuJava: an automated class mutation system , 2005 .

[15]  Matias Martinez,et al.  Fine-grained and accurate source code differencing , 2014, ASE.

[16]  Taher Ahmed Ghaleb,et al.  Large-Scale Manual Validation of Bug Fixing Commits: A Fine-grained Analysis of Tangling , 2020, ArXiv.

[17]  Xingyu Pan,et al.  CodEX: Source Code Plagiarism Detection Based on Abstract Syntax Tree , 2018, AICS.

[18]  J. Zobel,et al.  Mutation Testing for the New Century , 2001, The Springer International Series on Advances in Database Systems.

[19]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[20]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[21]  K. S. How Tai Wah.  Theoretical insights into the coupling effect , 2001 .

[22]  Yves Le Traon,et al.  Chapter Six - Mutation Testing Advances: An Analysis and Survey , 2019, Adv. Comput..

[23]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[24]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[25]  S. Herbold,et al.  Issues with SZZ: An empirical assessment of the state of practice of defect prediction data collection , 2019, ArXiv.

[26]  Alex Groce,et al.  Mutations: How Close are they to Real Faults? , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[27]  A. Jefferson Offutt,et al.  MuJava: an automated class mutation system , 2005, Softw. Test. Verification Reliab..