A Characteristic Study of Parameterized Unit Tests in .NET Open Source Projects

In the past decade, parameterized unit testing has emerged as a promising method to specify program behaviors under test in the form of unit tests. Developers can write parameterized unit tests (PUTs), unit-test methods with parameters, in contrast to conventional unit tests, without parameters. The use of PUTs can enable powerful test generation tools such as Pex to have strong test oracles to check against, beyond just uncaught runtime exceptions. In addition, PUTs have been popularly supported by various unit testing frameworks for .NET and the JUnit framework for Java. However, there exists no study to offer insights on how PUTs are written by developers in either proprietary or open source development practices, posing barriers for various stakeholders to bring PUTs to widely adopted practices in software industry. To fill this gap, we first present categorization results of the Microsoft MSDN Pex Forum posts (contributed primarily by industrial practitioners) related to PUTs. We then use the categorization results to guide the design of the first characteristic study of PUTs in .NET open source projects. We study hundreds of PUTs that open source developers wrote for these open source projects. Our study findings provide valuable insights for various stakeholders such as current or prospective PUT writers (e.g., developers), PUT framework designers, test-generation tool vendors, testing researchers, and testing educators.

[1]  Darko Marinov,et al.  An empirical analysis of flaky tests , 2014, SIGSOFT FSE.

[2]  David Saff.  Theory-infected: or how i learned to stop worrying and love universal quantification , 2007, OOPSLA '07.

[3]  Nikolai Tillmann,et al.  Parameterized unit tests , 2005, ESEC/FSE-13.

[4]  Dongmei Zhang,et al.  XIAO: tuning code clones at hands of engineers in practice , 2012, ACSAC '12.

[5]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[6]  Andreas Zeller,et al.  Generating parameterized unit tests , 2011, ISSTA '11.

[7]  C. A. R. HOARE,et al.  An axiomatic basis for computer programming , 1969, CACM.

[8]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[9]  Myra B. Cohen,et al.  Making system user interactive tests repeatable: when and what should we control? , 2015, ICSE 2015.

[10]  Dongmei Zhang,et al.  Transferring Code-Clone Detection and Analysis to Practice , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[11]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[12]  Nikolai Tillmann,et al.  Transferring an automated test generation tool to practice: from pex to fakes and code digger , 2014, ASE.

[13]  Nikolai Tillmann,et al.  Mutation Analysis of Parameterized Unit Tests , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[14]  Patrice Chalin,et al.  Are Practitioners Writing Contracts? , 2006, RODIN Book.

[15]  Andreas Podelski,et al.  If A Fails, Can B Still Succeed? Inferring Dependencies between Test Results in Automotive System Testing , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[16]  James J. Horning,et al.  The algebraic specification of abstract data types , 1978, Acta Informatica.

[17]  Nikolai Tillmann,et al.  Exploiting the synergy between automated-test-generation and programming-by-contract , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[18]  Nikolai Tillmann,et al.  Retrofitting Unit Tests for Parameterized Unit Testing , 2011, FASE.

[19]  Nikolai Tillmann,et al.  Parameterized unit testing: theory and practice , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[20]  Sebastian G. Elbaum,et al.  Test Analysis: Searching for Faults in Tests (N) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[21]  Bertrand Meyer,et al.  Applying 'design by contract' , 1992, Computer.

[22]  Albert L. Baker,et al.  Preliminary design of JML: a behavioral interface specification language for java , 2006, SOEN.

[23]  Bertrand Meyer,et al.  Contracts in Practice , 2012, FM.

[24]  Koushik Sen.  DART: Directed Automated Random Testing , 2009, Haifa Verification Conference.

[25]  Nikolai Tillmann,et al.  Precise identification of problems for structural test generation , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[26]  Michael D. Ernst,et al.  Case studies and tools for contract specifications , 2014, ICSE.