Supporting oracle construction via static analysis

In software testing, the program under test is usually executed with test inputs and checked against a test oracle, which is a mechanism to verify whether the program behaves as expected. Selecting the right oracle data to observe is crucial in test oracle construction. In the literature, researchers have proposed two dynamic approaches to oracle data selection by analyzing test execution information (e.g., variables' values or interaction information). However, collecting such information during program execution may incur extra cost. In this paper, we present the first static approach to oracle data selection, SODS (Static Oracle Data Selection). In particular, SODS first identifies the substitution relationships between candidate oracle data by constructing a probabilistic substitution graph based on the definition-use chains of the program under test, then estimates the fault-observing capability of each candidate oracle data, and finally selects a subset of oracle data with strong fault-observing capability. For programs with analyzable test code, we further extend SODS via pruning the probabilistic substitution graph based on 0-1-CFA call graph analysis. The experimental study on 11 subject systems written in C or Java demonstrates that our static approach is more effective and much more efficient than state-of-the-art dynamic approaches in most cases.

[1]  Debra J. Richardson,et al.  TAOS: Testing with Analysis and Oracle Support , 1994, ISSTA '94.

[2]  Gregg Rothermel,et al.  A Unified Test Case Prioritization Approach , 2014, ACM Trans. Softw. Eng. Methodol..

[3]  Paolo Tonella,et al.  Automated oracles: an empirical study on cost and effectiveness , 2013, ESEC/FSE 2013.

[4]  Phil McMinn,et al.  Search-based failure discovery using testability transformations to generate pseudo-oracles , 2009, GECCO.

[5]  Flemming Nielson,et al.  Principles of Program Analysis , 1999, Springer Berlin Heidelberg.

[6]  Qing Yu,et al.  Oracles for checking temporal properties of concurrent systems , 1994, SIGSOFT '94.

[7]  Alex Groce,et al.  Comparing non-adequate test suites using coverage criteria , 2013, ISSTA.

[8]  Shing-Chi Cheung,et al.  PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs , 2009, J. Syst. Softw..

[9]  Lu Zhang,et al.  Test Case Prioritization for Compilers: A Text-Vector Based Approach , 2016, 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[10]  Sarfraz Khurshid,et al.  Regression mutation testing , 2012, ISSTA 2012.

[11]  He Li,et al.  Test-Data Generation Guided by Static Defect Detection , 2009, Journal of Computer Science and Technology.

[12]  Lu Zhang,et al.  Search-based inference of polynomial metamorphic relations , 2014, ASE.

[13]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[14]  J. E. Freund,et al.  Modern Elementary Statistics , 1968 .

[15]  Lu Zhang,et al.  Inner oracles: input-specific assertions on internal states , 2015, ESEC/SIGSOFT FSE.

[16]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[17]  B. Eng,et al.  GENERATING A TEST ORACLE FROM PROGRAM DOCUMENTATION , 1995 .

[18]  Sarfraz Khurshid,et al.  Feedback-driven dynamic invariant discovery , 2014, ISSTA 2014.

[19]  Laura K. Dillon,et al.  Generating oracles from your favorite temporal logic specifications , 1996, SIGSOFT '96.

[20]  Klaus Havelund,et al.  Model Checking Programs , 2004, Automated Software Engineering.

[21]  Lu Zhang,et al.  An Empirical Comparison of Compiler Testing Techniques , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[22]  Michael D. Ernst,et al.  Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[23]  Huai Liu,et al.  How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? , 2014, IEEE Transactions on Software Engineering.

[24]  Michael D. Ernst,et al.  Dynamically discovering likely program invariants , 2000 .

[25]  Lu Zhang,et al.  Predictive Mutation Testing , 2016, IEEE Transactions on Software Engineering.

[26]  Mark C. Wilson,et al.  Introduction to Algorithms, Data Structures and Formal Languages, 2nd Edition , 2004 .

[27]  Raul H. C. Lopes,et al.  Pengaruh Latihan Small Sided Games 4 Lawan 4 Dengan Maksimal Tiga Sentuhan Terhadap Peningkatan VO2MAX Pada Siswa SSB Tunas Muda Bragang Klampis U-15 , 2022, Jurnal Ilmiah Mandala Education.

[28]  Jeffrey M. Voas,et al.  Putting assertions in their place , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[29]  Tao Xie,et al.  Mining test oracles of web search engines , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[30]  J. H. Andrews.  Deriving state-based test oracles for conformance testing , 2004 .

[31]  Tao Xie,et al.  Cooperative Software Testing and Analysis: Advances and Challenges , 2014, Journal of Computer Science and Technology.

[32]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[33]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[34]  René Just,et al.  MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler , 2011, 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011).

[35]  Lu Zhang,et al.  An Empirical Study on the Scalability of Selective Mutation Testing , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[36]  Barbara G. Ryder,et al.  Interprocedural Def-Use Associations for C Systems with Single Level Pointers , 1994, IEEE Trans. Software Eng..

[37]  Mats Per Erik Heimdahl,et al.  Programs, tests, and oracles: the foundations of testing revisited , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[38]  Andreas Zeller,et al.  Test generation across multiple layers , 2014, SBST 2014.

[39]  Shing-Chi Cheung,et al.  Reference Models and Automatic Oracles for the Testing of Mesh Simplification Software for Graphics Rendering , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[40]  Mario Piattini,et al.  Automated generation of test oracles using a model-driven approach , 2013, Inf. Softw. Technol..

[41]  David F. Bacon,et al.  Fast static analysis of C++ virtual function calls , 1996, OOPSLA '96.

[42]  David Grove,et al.  Call graph construction in object-oriented languages , 1997, OOPSLA '97.

[43]  Alexandru Nicolau,et al.  A language for conveying the aliasing properties of dynamic, pointer-based data structures , 1994, Proceedings of 8th International Parallel Processing Symposium.

[44]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[45]  Bogdan Korel,et al.  Generating expected results for automated black-box testing , 2002, Proceedings 17th IEEE International Conference on Automated Software Engineering,.

[46]  Michael D. Ernst,et al.  Are mutants a valid substitute for real faults in software testing? , 2014, SIGSOFT FSE.

[47]  Ronald L. Rivest,et al.  Introduction to Algorithms , 1990 .

[48]  Xin-She Yang,et al.  Introduction to Algorithms , 2021, Nature-Inspired Optimization Algorithms.

[49]  Gregory Gay,et al.  Automated Oracle Data Selection Support , 2015, IEEE Transactions on Software Engineering.

[50]  Gregg Rothermel,et al.  Dodona: automated oracle data set selection , 2014, ISSTA 2014.

[51]  Wei-Tek Tsai,et al.  An Approach for Service Composition and Testing for Cloud Computing , 2011, 2011 Tenth International Symposium on Autonomous Decentralized Systems.

[52]  Andreas Zeller,et al.  Checked coverage: an indicator for oracle quality , 2013, Softw. Test. Verification Reliab..

[53]  Massimiliano Di Penta,et al.  Assessing and improving state-based class testing: a series of experiments , 2004, IEEE Transactions on Software Engineering.

[54]  Gregg Rothermel,et al.  A Static Approach to Prioritizing JUnit Test Cases , 2012, IEEE Transactions on Software Engineering.

[55]  Leo Grady,et al.  Automating image segmentation verification and validation by learning test oracles , 2011, Inf. Softw. Technol..

[56]  Mary Lou Soffa,et al.  Efficient computation of interprocedural definition-use chains , 1994, TOPL.

[57]  Michael D. Ernst,et al.  Combined static and dynamic automated test generation , 2011, ISSTA '11.

[58]  Alessandra Gorla,et al.  Cross-checking oracles from intrinsic software redundancy , 2014, ICSE.

[59]  Nikolai Tillmann,et al.  DySy: dynamic symbolic execution for invariant inference , 2008, ICSE.

[60]  Gregory Gay,et al.  Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[61]  Michael D. Ernst,et al.  Eclat: Automatic Generation and Classification of Test Inputs , 2005, ECOOP.

[62]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[63]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[64]  David Kao,et al.  State coverage: a structural test adequacy criterion for behavior checking , 2007, ESEC-FSE companion '07.

[65]  Andreas Zeller,et al.  Assessing Oracle Quality with Checked Coverage , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[66]  Siti Zaiton Mohd Hashim,et al.  An automated framework for software test oracle , 2011, Inf. Softw. Technol..

[67]  David Grove,et al.  Optimization of Object-Oriented Programs Using Static Class Hierarchy Analysis , 1995, ECOOP.

[68]  David Lorge Parnas,et al.  Generating a test oracle from program documentation: work in progress , 1994, ISSTA '94.

[69]  Andreas Zeller,et al.  Mutation-Driven Generation of Unit Tests and Oracles , 2010, IEEE Transactions on Software Engineering.

[70]  Gordon Fraser,et al.  Does automated white-box test generation really help software testers? , 2013, ISSTA.