State Coverage: Software Validation Metrics beyond Code Coverage

Currently, testing is still the most important approach to reduce the amount of software defects. Software quality metrics help to prioritize where additional testing is necessary by measuring the quality of the code. Most approaches to estimate whether some unit of code is sufficiently tested are based on code coverage, which measures what code fragments are exercised by the test suite. Unfortunately, code coverage does not measure to what extent the test suite checks the intended functionality. We propose state coverage , a metric that measures the ratio of state updates that are read by assertions with respect to the total number of state updates, and we present efficient algorithms to measure state coverage. Like code coverage, state coverage is simple to understand and we show that it is effective to measure and easy to aggregate. During a preliminary evaluation on several open-source libraries, state coverage helped to identify multiple unchecked properties and detect several bugs.

[1]  Chris F. Kemerer,et al.  A Metrics Suite for Object Oriented Design , 2015, IEEE Trans. Software Eng..

[2]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[3]  Robert W. Floyd,et al.  Assigning Meanings to Programs , 1993 .

[4]  Alfred V. Aho,et al.  Data Structures and Algorithms , 1983 .

[5]  Anas N. Al-Rabadi,et al.  A comparison of modified reconstructability analysis and Ashenhurst‐Curtis decomposition of Boolean functions , 2004 .

[6]  Andreas Zeller,et al.  Javalanche: efficient mutation testing for Java , 2009, ESEC/SIGSOFT FSE.

[7]  Richard N. Taylor.  Assertions in programming languages , 1980, SIGP.

[8]  Alan Bundy,et al.  Constructing Induction Rules for Deductive Synthesis Proofs , 2006, CLASE.

[9]  David S. Rosenblum.  A Practical Approach to Programming With Assertions , 1995, IEEE Trans. Software Eng..

[10]  Gregory Tassey,et al.  Prepared for what , 2007 .

[11]  Nachiappan Nagappan,et al.  Assessing the Relationship between Software Assertions and Faults: An Empirical Investigation , 2006, 2006 17th International Symposium on Software Reliability Engineering.

[12]  Manuel Fähndrich,et al.  Embedded contract languages , 2010, SAC '10.

[13]  C. A. R. Hoare,et al.  Assertions: A Personal Perspective , 2003, IEEE Ann. Hist. Comput..

[14]  Mats Per Erik Heimdahl,et al.  Specification test coverage adequacy criteria = specification test generation inadequacy criteria , 2004, Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings..

[15]  David Kao,et al.  State coverage: a structural test adequacy criterion for behavior checking , 2007, ESEC-FSE companion '07.

[16]  Yong Rae Kwon,et al.  MuJava: an automated class mutation system: Research Articles , 2005 .

[17]  Frédéric Dadeau,et al.  Measuring a Java Test Suite Coverage Using JML Specifications , 2007, Electron. Notes Theor. Comput. Sci..

[18]  Banu Diri,et al.  A systematic review of software fault prediction studies , 2009, Expert Syst. Appl..

[19]  James C. King,et al.  Symbolic execution and program testing , 1976, CACM.

[20]  Tao Xie,et al.  UnitPlus: assisting developer testing in Eclipse , 2007, eclipse '07.

[21]  Sriram Sankar,et al.  Structural specification-based testing with ADL , 1996, ISSTA '96.

[22]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[23]  Andrew C. Myers,et al.  Language-based information-flow security , 2003, IEEE J. Sel. Areas Commun..

[24]  CatalCagatay,et al.  A systematic review of software fault prediction studies , 2009 .

[25]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[26]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..