Incrementally Discovering Testable Specifications from Program Executions

The ProTest project is an EU FP7 project to develop techniques that improve the testing and verification of concurrent and distributed software systems. One of the four main work packages is concerned with the automated identification of specifications that could serve as a suitable basis for testing; this is currently a tedious and error-prone manual task that tends to be neglected in practice. This paper describes how this problem has been addressed in the ProTest project. It describes a technique that uses test executions to refine the specification from which they are generated. It shows how the technique has been implemented and applied to real Erlang systems. It also describes in detail the major challenges that remain to be addressed in future work.

[1]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[2]  Neil Walkinshaw,et al.  Improving dynamic software analysis by applying grammar inference principles , 2008 .

[3]  Neil Walkinshaw,et al.  Inferring Finite-State Models with Temporal Constraints , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[4]  Pierre Dupont,et al.  Generating annotated behavior models from end-user scenarios , 2005, IEEE Transactions on Software Engineering.

[5]  K. H. Bennett,et al.  Journal of software maintenance : research and practice , 1989 .

[6]  John Derrick,et al.  Iterative Refinement of Reverse-Engineered Models by Model-Based Testing , 2009, FM.

[7]  Hod Lipson,et al.  Nonlinear system identification using coevolution of models and tests , 2005, IEEE Transactions on Evolutionary Computation.

[8]  Alexander L. Wolf,et al.  Discovering models of software processes from event-based data , 1998, TSEM.

[9]  Roland Groz,et al.  Inferring Mealy Machines , 2009, FM.

[10]  Joe Armstrong,et al.  Programming Erlang: Software for a Concurrent World , 1993 .

[11]  Alan W. Biermann,et al.  Constructing Programs from Example Computations , 1976, IEEE Transactions on Software Engineering.

[12]  William G. Griswold,et al.  Dynamically discovering likely program invariants to support program evolution , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[13]  Thomas Arts,et al.  Automatic testing of TCP/IP implementations using QuickCheck , 2009, Erlang Workshop.

[14]  Sheeva Afshan,et al.  Using compression algorithms to support the comprehension of program traces , 2010, WODA '10.

[15]  Neil Walkinshaw,et al.  Automated discovery of state transitions and their functions in source code , 2008 .

[16]  Jerome A. Feldman,et al.  On the Synthesis of Finite-State Machines from Samples of Their Behavior , 1972, IEEE Transactions on Computers.

[17]  Lars-Åke Fredlund,et al.  Property-Based Testing - The ProTest Project , 2009, FMCO.

[18]  Stefan C. Kremer,et al.  Inducing Grammars from Sparse Data Sets: A Survey of Algorithms and Results , 2003, J. Mach. Learn. Res..

[19]  Koen Claessen,et al.  QuickCheck: a lightweight tool for random testing of Haskell programs , 2011, SIGP.

[20]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[21]  A. Nerode,et al.  Linear automaton transformations , 1958 .

[22]  Edward F. Moore,et al.  Gedanken-Experiments on Sequential Machines , 1956 .

[23]  Ana Cavalcanti,et al.  FM 2009: Formal Methods, Second World Congress, Eindhoven, The Netherlands, November 2-6, 2009. Proceedings , 2009, FM.

[24]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[25]  Walter F. Tichy,et al.  Proceedings 25th International Conference on Software Engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[26]  Sheeva Afshan,et al.  Using Dictionary Compression Algorithms to Identify Phases in Program Traces , 2010 .

[27]  Pierre Dupont,et al.  THE QSM ALGORITHM AND ITS APPLICATION TO SOFTWARE BEHAVIOR MODEL INDUCTION , 2008, Appl. Artif. Intell..

[28]  Terry A. Welch,et al.  A Technique for High-Performance Data Compression , 1984, Computer.

[29]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[30]  Kwang-Ting Cheng,et al.  Automatic Functional Test Generation Using The Extended Finite State Machine Model , 1993, 30th ACM/IEEE Design Automation Conference.

[31]  Jon Postel,et al.  Transmission Control Protocol , 1981, RFC.

[32]  Arie van Deursen,et al.  Execution trace analysis through massive sequence and circular bundle views , 2008, J. Syst. Softw..

[33]  Neil Walkinshaw,et al.  Reverse Engineering State Machines by Interactive Grammar Inference , 2007, 14th Working Conference on Reverse Engineering (WCRE 2007).

[34]  Craig G. Nevill-Manning,et al.  Compression and Explanation Using Hierarchical Grammars , 1997, Comput. J..

[35]  James R. Larus,et al.  Mining specifications , 2002, POPL '02.

[36]  Barak A. Pearlmutter,et al.  Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm , 1998, ICGI.