Hunting for smells in natural language tests

Tests are central artifacts of software systems and play a crucial role for software quality. In system testing, a lot of test execution is performed manually using tests in natural language. However, those test cases are often poorly written without best practices in mind. This leads to tests which are not maintainable, hard to understand and inefficient to execute. For source code and unit tests, so called code smells and test smells have been established as indicators to identify poorly written code. We apply the idea of smells to natural language tests by defining a set of common Natural Language Test Smells (NLTS). Furthermore, we report on an empirical study analyzing the extent in more than 2800 tests of seven industrial test suites.

[1]  Bernhard Schätz,et al.  Can clone detection support quality assessments of requirements specifications? , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[2]  Arie van Deursen,et al.  Refactoring test code , 2001 .

[3]  Gerard Meszaros,et al.  The Test Automation Manifesto , 2003, XP/Agile Universe.

[4]  Foutse Khomh,et al.  An Exploratory Study of the Impact of Code Smells on Software Change-proneness , 2009, 2009 16th Working Conference on Reverse Engineering.

[5]  Elmar Jürgens,et al.  Can clone detection support test comprehension? , 2012, 2012 20th IEEE International Conference on Program Comprehension (ICPC).

[6]  Paolo Tonella,et al.  Lexicon Bad Smells in Software , 2009, 2009 16th Working Conference on Reverse Engineering.

[7]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[8]  Foutse Khomh,et al.  An Empirical Study of the Impact of Two Antipatterns, Blob and Spaghetti Code, on Program Comprehension , 2011, 2011 15th European Conference on Software Maintenance and Reengineering.

[9]  Olga Ormandjieva,et al.  Automatic Quality Assessment of SRS Text by Means of a Decision-Tree-Based Text Classifier , 2007, Seventh International Conference on Quality Software (QSIC 2007).

[10]  Linda H. Rosenberg,et al.  Automated Analysis of Requirement Specifications , 1997, Proceedings of the (19th) International Conference on Software Engineering.

[11]  Gerard Meszaros,et al.  xUnit Test Patterns: Refactoring Test Code , 2007 .

[12]  Stefania Gnesi,et al.  An Automatic Quality Evaluation for Natural Language Requirements , 2001 .

[13]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[14]  Reinhold Plösch,et al.  The Quamoco product quality modelling and assessment approach , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[15]  Gabriele Bavota,et al.  An empirical analysis of the distribution of unit test smells and their impact on software maintenance , 2012, 2012 28th IEEE International Conference on Software Maintenance (ICSM).

[16]  Arie van Deursen,et al.  The Video Store Revisited – Thoughts on Refactoring and Testing , 2002 .