Practitioners' Views on Good Software Testing Practices

Software testing is an integral part of software development process. Unfortunately, for many projects, bugs are prevalent despite testing effort, and testing continues to cost significant amount of time and resources. This brings forward the issue of test case quality and prompts us to investigate what make good test cases. To answer this important question, we interview 21 and survey 261 practitioners, who come from many small to large companies and open source projects distributed in 27 countries, to create and validate 29 hypotheses that describe characteristics of good test cases and testing practices. These characteristics span multiple dimensions including test case contents, size and complexity, coverage, maintainability, and bug detection. We present highly rated characteristics and rationales why practitioners agree or disagree with them, which in turn highlight best practices and trade-offs that need to be considered in the creation of test cases. Our findings also highlight open problems and opportunities for software engineering researchers to improve practitioner activities and address their pain points.

[1]  Ramanath Subramanyam,et al.  Empirical Analysis of CK Metrics for Object-Oriented Design Complexity: Implications for Software Defects , 2003, IEEE Trans. Software Eng..

[2]  Andy Zaidman,et al.  Does Refactoring of Test Smells Induce Fixing Flaky Tests? , 2017, 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[3]  Gordon Fraser,et al.  A Survey on Unit Testing Practices and Problems , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[4]  Feng Zhu,et al.  Software Internationalization and Localization: An Industrial Experience , 2013, 2013 18th International Conference on Engineering of Complex Computer Systems.

[5]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[6]  Serge Demeyer,et al.  On The Detection of Test Smells: A Metrics-Based Approach for General Fixture and Eager Test , 2007, IEEE Trans. Software Eng..

[7]  Neil A. Ernst,et al.  Measure it? Manage it? Ignore it? software practitioners and technical debt , 2015, ESEC/SIGSOFT FSE.

[8]  Zhenchang Xing,et al.  Experience report: An industrial experience report on test outsourcing practices , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[9]  Michael D. Ernst,et al.  Automatic test factoring for java , 2005, ASE '05.

[10]  Jane Cleland-Huang,et al.  Towards more intelligent trace retrieval algorithms , 2014, RAISE 2014.

[11]  Boyang Li,et al.  Automatically Documenting Unit Test Cases , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[12]  David Lo,et al.  Understanding the Test Automation Culture of App Developers , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[13]  David Lo,et al.  Code coverage and test suite effectiveness: Empirical study with real bugs in large systems , 2015, 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[14]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[15]  Alex Groce,et al.  Code coverage for suite evaluation by developers , 2014, ICSE.

[16]  Jurriaan Hage,et al.  How do professionals perceive legacy systems and software modernization? , 2014, ICSE.

[17]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[18]  Rjg Raymond Opdenakker.  Advantages and disadvantages of four interview techniques in qualitative research , 2006 .

[19]  Yucheng Zhang,et al.  Assertions are strongly correlated with test suite effectiveness , 2015, ESEC/SIGSOFT FSE.

[20]  Arie van Deursen,et al.  Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining , 2008, Empirical Software Engineering.

[21]  Arie van Deursen,et al.  Automated Detection of Test Fixture Strategies and Smells , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.

[22]  Premkumar T. Devanbu,et al.  Belief & Evidence in Empirical Software Engineering , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[23]  Matthew B. Dwyer,et al.  Carving differential unit test cases from system test cases , 2006, SIGSOFT '06/FSE-14.

[24]  David Lo,et al.  Customer satisfaction feedback in an IT outsourcing company: a case study on the insigma Hengtian company , 2015, EASE.

[25]  Premkumar T. Devanbu,et al.  A large scale study of programming languages and code quality in github , 2014, SIGSOFT FSE.

[26]  Gregory Tassey,et al.  Prepared for what , 2007 .

[27]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[28]  Miryung Kim,et al.  The Emerging Role of Data Scientists on Software Development Teams , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[29]  Amin Milani Fard,et al.  An empirical study of bugs in test code , 2015, 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME).

[30]  Christopher B. Mayhorn,et al.  Quantifying developers' adoption of security tools , 2015, ESEC/SIGSOFT FSE.