Black-Box Test Generation from Inferred Models

Automatically generating test inputs for components without source code (are 'black-box') and specification is challenging. One particularly interesting solution to this problem is to use Machine Learning algorithms to infer testable models from program executions in an iterative cycle. Although the idea has been around for over 30 years, there is little empirical information to inform the choice of suitable learning algorithms, or to show how good the resulting test sets are. This paper presents an openly available framework to facilitate experimentation in this area, and provides a proof-of-concept inference-driven testing framework, along with evidence of the efficacy of its test sets on three programs.

[1]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[2]  John Derrick,et al.  Increasing Functional Coverage by Inductive Testing: A Case Study , 2010, ICTSS.

[3]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[4]  Hong Zhu.  A Formal Interpretation of Software Testing as Inductive Inference , 1996 .

[5]  Lionel C. Briand,et al.  Using machine learning to refine Category-Partition test specifications and test suites , 2009, Inf. Softw. Technol..

[6]  John A. Clark,et al.  Strengthening Inferred Specifications using Search Based Testing , 2008, 2008 IEEE International Conference on Software Testing Verification and Validation Workshop.

[7]  Jeffrey Scott Vitter,et al.  Using Vapnik-Chervonenkis Dimension to Analyze the Testing Complexity of Program Segments , 1996, Inf. Comput..

[8]  Gordon Fraser,et al.  Behaviourally Adequate Software Testing , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[9]  Hong Zhu,et al.  Inductive inference and software testing , 1992, Softw. Test. Verification Reliab..

[10]  金田 重郎,et al.  C4.5: Programs for Machine Learning (書評) , 1995 .

[11]  Roland Groz,et al.  Inferring Mealy Machines , 2009, FM.

[12]  Dana Angluin,et al.  Two notions of correctness and their relation to testing , 1982, Acta Informatica.

[13]  Elaine J. Weyuker,et al.  Assessing Test Data Adequacy through Program Inference , 2019, TOPL.

[14]  Neil Walkinshaw,et al.  Assessing Test Adequacy for Black-Box Systems without Specifications , 2011, ICTSS.

[15]  Karl Meinke,et al.  Incremental Learning-Based Testing for Reactive Systems , 2011, TAP@TOOLS.

[16]  Harald Raffelt,et al.  LearnLib: a library for automata learning and experimentation , 2005, FMICS '05.

[17]  John Derrick,et al.  Iterative Refinement of Reverse-Engineered Models by Model-Based Testing , 2009, FM.

[18]  Francesco Bergadano,et al.  Testing by means of inductive program learning , 1996, TSEM.

[19]  Yue Jia,et al.  MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language , 2008, Testing: Academic & Industrial Conference - Practice and Research Techniques (taic part 2008).

[20]  Carl H. Smith,et al.  A Recursion Theoretic Approach to Program Testing , 1987, IEEE Transactions on Software Engineering.

[21]  David Cohn,et al.  Active Learning , 2010, Encyclopedia of Machine Learning.

[22]  Gregg Rothermel,et al.  Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact , 2005, Empirical Software Engineering.

[23]  Hong Zhu,et al.  A Formal Interpretation of Software Testing as Inductive Inference , 1996, Softw. Test. Verification Reliab..

[24]  Kathleen Romanik.  Approximate Testing and Its Relationship to Learning , 1997, Theor. Comput. Sci..