Intelligent Test Oracle Construction for Reactive Systems without Explicit Specifications

A test oracle is a mechanism that decides whether an SUT (software under test) fails or passes a test case. We investigate how to use machine learning techniques to automatically construct test oracles for reactive programs without reliance on explicit specifications. Firstly, we present a library, called In TOL (Intelligent Test Oracle Library), for the convenient and flexible collection of test traces. We can flexibly use either user guidance or program assertions to collect verdicts to test traces. Such verdicts are used as supervisory signals to the supervised learning algorithm (SLA) for a test oracle. Secondly, we present several sets of feature variables for the temporal relation among events in test traces of unbounded lengths. Then we present procedures that convert test traces into feature vectors, train an SLA with the feature vectors and their verdicts, and use the trained SLA as a test oracle. We report the implementation of In TOL on top of SVM (support vector machine). We experiment with two open-source benchmark SUTs from the internet to check the performance of our techniques. Our experiment data shows that high-accuracy test verdicts can be achieved with our test oracles for the benchmark SUTs.

[1]  David Casasent,et al.  A hierarchical classifier using new support vector machine , 2005, Eighth International Conference on Document Analysis and Recognition (ICDAR'05).

[2]  Siti Zaiton Mohd Hashim,et al.  A Comparative Study on Automated Software Test Oracle Methods , 2009, 2009 Fourth International Conference on Software Engineering Advances.

[3]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[4]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[5]  Petar Tahchiev.  JUnit in action , 2011 .

[6]  Fred Kröger,et al.  Temporal Logic of Programs , 1987, EATCS Monographs on Theoretical Computer Science.

[7]  K. K. Aggarwal,et al.  A neural net based approach to Test Oracle , 2004, SOEN.

[8]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[9]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[10]  Abraham Kandel,et al.  Using a neural network in the software testing process , 2002, Int. J. Intell. Syst..

[11]  Matthias Kalle Dalheimer,et al.  Programming with Qt , 1999 .

[12]  David Lorge Parnas,et al.  Generating a test oracle from program documentation: work in progress , 1994, ISSTA '94.

[13]  Edmund M. Clarke,et al.  Design and Synthesis of Synchronization Skeletons Using Branching Time Temporal Logic , 2008, 25 Years of Model Checking.

[14]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[15]  Farn Wang,et al.  Evolving a Test Oracle in Black-Box Testing , 2011, FASE.

[16]  M. W. Shields.  An Introduction to Automata Theory , 1988 .

[17]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[18]  Mao Ye,et al.  Automated Test Oracle Based on Neural Networks , 2006, 2006 5th IEEE International Conference on Cognitive Informatics.

[19]  Chih-Jen Lin,et al.  Working Set Selection Using Second Order Information for Training Support Vector Machines , 2005, J. Mach. Learn. Res..

[20]  Mary Lou Soffa,et al.  Automated test oracles for GUIs , 2000, SIGSOFT '00/FSE-8.

[21]  B. Eng,et al.  GENERATING A TEST ORACLE FROM PROGRAM DOCUMENTATION , 1995 .