How well are regular expressions tested in the wild?

Developers report testing their regular expressions less than the rest of their code. In this work, we explore how thoroughly tested regular expressions are by examining open source projects. Using standard metrics of coverage, such as line and branch coverage, gives an incomplete picture of the test coverage of regular expressions. We adopt graph-based coverage metrics for the DFA representation of regular expressions, providing fine-grained test coverage metrics. Using over 15,000 tested regular expressions in 1,225 Java projects on GitHub, we measure node, edge, and edge-pair coverage. Our results show that only 17% of the regular expressions in the repositories are tested at all. For those that are tested, the median number of test inputs is two. For nearly 42% of the tested regular expressions, only one test input is used. Average node and edge coverage levels on the DFAs for tested regular expressions are 59% and 29%, respectively. Due to the lack of testing of regular expressions, we explore whether a string generation tool for regular expressions, Rex, achieves high coverage levels. With some exceptions, we found that tools such as Rex can be used to write test inputs with similar coverage to the developer tests.

[1]  Michael D. Ernst,et al.  Randoop: feedback-directed random testing for Java , 2007, OOPSLA '07.

[2]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[3]  Paolo Arcaini,et al.  MutRex: A Mutation-Based Generator of Fault Detecting Strings for Regular Expressions , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).

[4]  Jeffrey E. F. Friedl.  Mastering Regular Expressions , 1997 .

[5]  David Notkin,et al.  Symstra: A Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution , 2005, TACAS.

[6]  Gordon Fraser,et al.  Does automated white-box test generation really help software testers? , 2013, ISSTA.

[7]  Kathryn T. Stolee,et al.  Exploring regular expression usage and context in Python , 2016, ISSTA.

[8]  Shigeru Chiba.  Javassist - A Reflection-based Programming Wizard for Java , 1998 .

[9]  Michael D. Ernst,et al.  HAMPI: a solver for string constraints , 2009, ISSTA.

[10]  Nikolai Tillmann,et al.  Reggae: Automated Test Generation for Programs Using Complex Regular Expressions , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[11]  Margus Veanes,et al.  Rex: Symbolic Regular Expression Explorer , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[12]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[13]  Eric Larson,et al.  Generating Evil Test Strings for Regular Expressions , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[14]  Michael D. Ernst,et al.  HAMPI: A solver for word equations over strings, regular expressions, and context-free grammars , 2012, TSEM.

[15]  Meiyappan Nagappan,et al.  Curating GitHub for engineered software projects , 2017, Empirical Software Engineering.

[16]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[17]  Michael O'Neill,et al.  A search for improved performance in regular expressions , 2017, GECCO.

[18]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[19]  Michael Sipser,et al.  Introduction to the Theory of Computation , 1996, SIGA.

[20]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[21]  Michael D. Ernst,et al.  Combined static and dynamic automated test generation , 2011, ISSTA '11.

[22]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[23]  Sriram Raghavan,et al.  Regular Expression Learning for Information Extraction , 2008, EMNLP.

[24]  Fabian Beck,et al.  RegViz: visual debugging of regular expressions , 2014, ICSE Companion.

[25]  James M. Bieman,et al.  Software reliability growth with test coverage , 2002, IEEE Trans. Reliab..

[26]  Rohit Babbar,et al.  Clustering based approach to learning regular expressions over large alphabet for noisy unstructured text , 2010, AND '10.

[27]  George Candea,et al.  Parallel symbolic execution for automated real-world software testing , 2011, EuroSys '11.

[28]  Guodong Li,et al.  JST: An automatic test generation tool for industrial Java applications with strings , 2013, 2013 35th International Conference on Software Engineering (ICSE).

[29]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[30]  Hiroshi Inamura,et al.  Dynamic test input generation for web applications , 2008, ISSTA '08.

[31]  Kathryn T. Stolee,et al.  Exploring regular expression comprehension , 2017, 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE).

[32]  Paul Piwowarski,et al.  Coverage measurement experience during function test , 1993, Proceedings of 1993 15th International Conference on Software Engineering.

[33]  A. Jefferson Offutt,et al.  An Experimental Comparison of Four Unit Test Criteria: Mutation, Edge-Pair, All-Uses and Prime Path Coverage , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.