Generic and Effective Specification of Structural Test Objectives

A large amount of research has been carried out to automate white-box testing. While a wide range of different and sometimes heterogeneous code-coverage criteria have been proposed, there exists no generic formalism to describe them all, and available test automation tools usually support only a small subset of them. We introduce a new specification language, called HTOL (Hyperlabel Test Objectives Language), providing a powerful generic mechanism to define a wide range of test objectives. HTOL comes with a formal semantics, and can encode all standard criteria but full mutations. Besides specification, HTOL is appealing in the context of test automation as it allows handling criteria in a unified way.

[1]  Tsong Yueh Chen,et al.  Test case selection strategies based on Boolean specifications ‡ , 2001, Softw. Test. Verification Reliab..

[2]  Zohar Manna,et al.  The Temporal Logic of Reactive and Concurrent Systems , 1991, Springer New York.

[3]  Richard J. Lipton,et al.  Hints on Test Data Selection: Help for the Practicing Programmer , 1978, Computer.

[4]  Alex Groce,et al.  TSTL: a language and tool for testing (demo) , 2015, ISSTA.

[5]  Anna Philippou,et al.  Tools and Algorithms for the Construction and Analysis of Systems , 2018, Lecture Notes in Computer Science.

[6]  Kuo-Chung Tai,et al.  Theory of Fault-Based Predicate Testing for Computer Programs , 1996, IEEE Trans. Software Eng..

[7]  Miodrag Potkonjak,et al.  MediaBench: a tool for evaluating and synthesizing multimedia and communications systems , 1997, Proceedings of 30th Annual International Symposium on Microarchitecture.

[8]  Johannes Kinder,et al.  Hypertesting : The Case for Automated Testing of Hyperproperties , 2015 .

[9]  Sarfraz Khurshid,et al.  Test generation through programming in UDITA , 2010, 2010 ACM/IEEE 32nd International Conference on Software Engineering.

[10]  Yves Le Traon,et al.  Sound and Quasi-Complete Detection of Infeasible Test Requirements , 2015, 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST).

[11]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[12]  Marsha Chechik,et al.  A buffer overflow benchmark for software model checkers , 2007, ASE.

[13]  Dieter Hogrefe,et al.  An introduction to the testing and test control notation (TTCN-3) , 2003, Comput. Networks.

[14]  Bruno Marre,et al.  On-the-fly generation of k-path tests for C functions , 2004, Proceedings. 19th International Conference on Automated Software Engineering, 2004..

[15]  A. Jefferson Offutt,et al.  Evaluation of three specification-based testing criteria , 2000, Proceedings Sixth IEEE International Conference on Engineering of Complex Computer Systems. ICECCS 2000.

[16]  Mario Friske,et al.  Composition of Model-based Test Coverage Criteria , 2008, MBEES.

[17]  Bengt Jonsson,et al.  Specifying and Generating Test Cases Using Observer Automata , 2004, FATES.

[18]  Kim G. Larsen,et al.  Testing Real-Time Systems Using UPPAAL , 2008, Formal Methods and Testing.

[19]  Nikolai Tillmann,et al.  Guided test generation for coverage criteria , 2010, 2010 IEEE International Conference on Software Maintenance.

[20]  Insup Lee,et al.  A Temporal Logic Based Theory of Test Coverage and Generation , 2002, TACAS.

[21]  Sharad Malik,et al.  Computer Aided Verification, 20th International Conference, CAV 2008, Princeton, NJ, USA, July 7-14, 2008, Proceedings , 2008, CAV.

[22]  Janusz W. Laski,et al.  A Data Flow Oriented Program Testing Strategy , 1983, IEEE Transactions on Software Engineering.

[23]  Marius Mikucionis,et al.  Formal Methods and Testing , 2008 .

[24]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[25]  Elaine J. Weyuker,et al.  A Formal Analysis of the Fault-Detecting Ability of Testing Methods , 1993, IEEE Trans. Software Eng..

[26]  Ina Schieferdecker,et al.  The UML 2.0 Testing Profile and Its Relation to TTCN-3 , 2003, TestCom.

[27]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[28]  Lori A. Clarke,et al.  A Formal Model of Program Dependences and Its Implications for Software Testing, Debugging, and Maintenance , 1990, IEEE Trans. Software Eng..

[29]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[30]  RothermelGregg,et al.  Supporting Controlled Experimentation with Testing Techniques , 2005 .

[31]  Bernd Finkbeiner,et al.  Temporal Logics for Hyperproperties , 2013, POST.

[32]  Helmut Veith,et al.  FShell: Systematic Test Case Generation for Dynamic Analysis and Measurement , 2008, CAV.

[33]  Qian Yang,et al.  A survey of coverage based testing tools , 2006, AST '06.

[34]  Michael R. Clarkson,et al.  Hyperproperties , 2008, 2008 21st IEEE Computer Security Foundations Symposium.

[35]  Mark Ryan,et al.  Reduction of Equational Theories for Verification of Trace Equivalence: Re-encryption, Associativity and Commutativity , 2012, POST.

[36]  Nikolai Kosmatov,et al.  An All-in-One Toolkit for Automated White-Box Testing , 2014, TAP@STAF.

[37]  Jonathan P. Bowen,et al.  From MC/DC to RC/DC: formalization and analysis of control-flow testing criteria , 2006, Formal Aspects of Computing.

[38]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[39]  Sanjai Rayadurgam,et al.  Generating MC/DC adequate test sequences through model checking , 2003, 28th Annual NASA Goddard Software Engineering Workshop, 2003. Proceedings..

[40]  A. Jefferson Offutt,et al.  An Empirical Evaluation of Weak Mutation , 1994, IEEE Trans. Software Eng..

[41]  A. Jefferson Offutt,et al.  Coverage criteria for logical expressions , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[42]  Aditya P. Mathur,et al.  Foundations of Software Testing , 2007 .

[43]  William E. Howden,et al.  Weak Mutation Testing and Completeness of Test Sets , 1982, IEEE Transactions on Software Engineering.

[44]  Rajeev Alur,et al.  A Temporal Logic of Nested Calls and Returns , 2004, TACAS.

[45]  Nikolai Kosmatov,et al.  Taming Coverage Criteria Heterogeneity with LTest , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[46]  Helmut Veith,et al.  How did you specify your test suite , 2010, ASE.

[47]  Nikolai Kosmatov,et al.  Efficient Leveraging of Symbolic Execution to Advanced Coverage Criteria , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation.

[48]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..