Classifying Test Suite Effectiveness via Model Inference and ROBBDs

Deciding whether a given test suite is effective enough is certainly a challenging task. Focusing on a software program’s functionality, we propose in this paper a new method that leverages Boolean functions as abstract reasoning format. That is, we use machine learning in order to infer a special binary decision diagram from the considered test suite and extract a total variable order, if possible. Intuitively, if an ROBDD derived from the Boolean functions representing the program under test’s specification actually coincides with that of the test suite (using the same variable order), we conclude that the test suite is effective enough. That is, any program that passes such a test suite should clearly show the desired input-output behavior. In our paper, we provide the corresponding algorithms of our approach and their respective proofs. Our first experimental results illustrate our approach’s practicality and viability.

[1]  Elaine J. Weyuker,et al.  Automatically Generating Test Data from a Boolean Specification , 1994, IEEE Trans. Software Eng..

[2]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[3]  Elaine J. Weyuker,et al.  Assessing Test Data Adequacy through Program Inference , 2019, TOPL.

[4]  Shaul Markovitch,et al.  Learning to Order BDD Variables in Verification , 2011, J. Artif. Intell. Res..

[5]  Neil Walkinshaw.  The Practical Assessment of Test Sets with Inductive Inference Techniques , 2010, TAIC PART.

[6]  Claude E. Shannon,et al.  The synthesis of two-terminal switching circuits , 1949, Bell Syst. Tech. J..

[7]  Yves Le Traon,et al.  MutaLog: A Tool for Mutating Logic Formulas , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[8]  John Derrick,et al.  Inferring extended finite state machine models from software executions , 2013, 2013 20th Working Conference on Reverse Engineering (WCRE).

[9]  Kenneth J. Supowit,et al.  Finding the Optimal Variable Ordering for Binary Decision Diagrams , 1987, 24th ACM/IEEE Design Automation Conference.

[10]  Man Fai Lau,et al.  Redefinition of Fault Classes in Logic Expressions , 2012, 2012 12th International Conference on Quality Software.

[11]  Randal E. Bryant,et al.  Symbolic Boolean manipulation with ordered binary-decision diagrams , 1992, CSUR.

[12]  Sheldon B. Akers,et al.  Binary Decision Diagrams , 1978, IEEE Transactions on Computers.

[13]  E. Clarke,et al.  Multi-Terminal Binary Decision Diagrams and Hybrid Decision Diagrams , 1996 .

[14]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[15]  Roland Groz,et al.  Inferring Mealy Machines , 2009, FM.

[16]  Hanspeter Mössenböck,et al.  Single-pass generation of static single-assignment form for structured languages , 1994, TOPL.

[17]  Karl Meinke,et al.  Incremental Learning-Based Testing for Reactive Systems , 2011, TAP@TOOLS.

[18]  Bernhard Steffen,et al.  Introduction to Active Automata Learning from a Practical Perspective , 2011, SFM.

[19]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[20]  Robert E. Tarjan,et al.  Depth-First Search and Linear Graph Algorithms , 1972, SIAM J. Comput..

[21]  Tsong Yueh Chen,et al.  MUMCUT: a fault-based strategy for testing Boolean specifications , 1999, Proceedings Sixth Asia Pacific Software Engineering Conference (ASPEC'99) (Cat. No.PR00509).

[22]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[23]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.