An empirical study on the effects of code visibility on program testability

Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an abstract property of programs, testability can be measured by various metrics, which are affected by different factors of design and implementation. In object-oriented software development, code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has some effects on testability. However, little empirical evidence has been shown to clarify whether and how software testability is influenced by code visibility. We have performed a comprehensive empirical study to shed light on this problem. We first use code coverage as a concrete proxy for testability. We select 27 real-world software programs as subjects and ran two state-of-the-art automated testing tools, Randoop and EvoSuite, on these programs to analyze their code coverage, in comparison with that of developer-written tests. The results show that code visibility does not necessarily have effects on code coverage, but can significantly affect automated tools. Developer-written tests achieve similar coverage on code areas with different visibility, while low code visibility often leads to low code coverage for automated tools. In addition, we have developed two enhanced variants of Randoop that implement multiple strategies to handle code visibility. The results on Randoop variants show that different treatments on code visibility can result in significant differences in code coverage for automated tools. In the second part, our study uses fault detection rate as another concrete measurement of testability. We apply the automated testing tools on 357 real faults. The result of our in-depth analysis is consistent with that of the first part, demonstrating the significant effects of code visibility on program testability.

[1]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[2]  Hiroyuki Sato,et al.  An Empirical Study on Effects of Code Visibility on Code Coverage of Software Testing , 2015, 2015 IEEE/ACM 10th International Workshop on Automation of Software Test.

[3]  Stefan Jungmayr.  Design for Testability , 2002 .

[4]  Hiroyuki Sato,et al.  GRT at the SBST 2015 Tool Competition , 2015, 2015 IEEE/ACM 8th International Workshop on Search-Based Software Testing.

[5]  Michael D. Ernst,et al.  Defects4J: a database of existing faults to enable controlled testing studies for Java programs , 2014, ISSTA 2014.

[6]  Guy L. Steele,et al.  The Java Language Specification, Java SE 8 Edition , 2013 .

[7]  Corina S. Pasareanu,et al.  JPF-SE: A Symbolic Execution Extension to Java PathFinder , 2007, TACAS.

[8]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[9]  Kentaro Ishii,et al.  Pebbles: User-Configurable Device Network for Robot Navigation , 2013, INTERACT.

[10]  Stefan Jungmayr.  Reviewing Software Artifacts for Testability , 1999 .

[11]  Cédric Beust,et al.  Next Generation Java Testing: TestNG and Advanced Concepts , 2007 .

[12]  Cyrille Artho,et al.  Classification of Randomly Generated Test Cases , 2016, 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER).

[13]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[14]  Roger T. Alexander,et al.  Design-for-Testability for Object-Oriented Software 1 , 1997 .

[15]  Carl K. Chang,et al.  Practical Extensions of a Randomized Testing Tool , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[16]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[17]  Reid Holmes,et al.  Coverage is not strongly correlated with test suite effectiveness , 2014, ICSE.

[18]  Guy L. Steele,et al.  The Java Language Specification , 1996 .

[19]  Timothy A. Budd,et al.  An introduction to object-oriented programming , 1991 .

[20]  Gerald M. Salzman.  An Introduction to Object Oriented Programming , 2013 .

[21]  Lei Ma,et al.  Retrofitting automatic testing through library tests reusing , 2016, 2016 IEEE 24th International Conference on Program Comprehension (ICPC).

[22]  M. Young.  Test Oracles , 2001 .

[23]  Hiroyuki Sato,et al.  GRT: An Automated Test Generator Using Orchestrated Program Analysis , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[24]  Magiel Bruntink,et al.  Testability of Object-Oriented Systems: a Metrics-based Approach , 2003 .

[25]  Hiroyuki Sato,et al.  Efficient testing of software product lines via centralization (short paper) , 2014, GPCE.

[26]  GORDON FRASER,et al.  A Large-Scale Evaluation of Automated Unit Test Generation Using EvoSuite , 2014, ACM Trans. Softw. Eng. Methodol..

[27]  Lei Ma,et al.  Incremental Web Application Testing Using Page Object , 2015, 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb).

[28]  Vincent Massol,et al.  JUnit in Action , 2003 .

[29]  Gregory Gay,et al.  Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[30]  Hiroyuki Sato,et al.  GRT: Program-Analysis-Guided Random Testing (T) , 2015, 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).

[31]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[32]  Robert V. Binder,et al.  Design for testability in object-oriented systems , 1994, CACM.

[33]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[34]  Javam C. Machado,et al.  The prediction of faulty classes using object-oriented design metrics , 2001, J. Syst. Softw..