A framework for the competitive evaluation of model inference techniques

This paper describes the STAMINA competition1, which is designed to drive the evaluation and improvement of software model-inference approaches. To this end, the target models have certain characteristics that tend to appear in software-models; they have large alphabets, and states are not evenly connected by transitions (as has been the case in previous similar competitions). The paper describes the set-up of the competition that extends previous similar competitions in the field of regular grammar inference. However, this competition focusses on target models that are characteristic of software systems, and features a suitably adapted protocol for the generation of training and testing samples. Besides providing details of the competition itself, it also discusses how outcomes from the competition will be used to gain broader insights into the relative accuracy and efficiency of competing techniques.

[1]  Neil Walkinshaw,et al.  Computing the Structural Difference between State-Based Models , 2009, 2009 16th Working Conference on Reverse Engineering.

[2]  Siau-Cheng Khoo,et al.  SMArTIC: towards building an accurate, robust and scalable specification miner , 2006, SIGSOFT '06/FSE-14.

[3]  Barak A. Pearlmutter,et al.  Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm , 1998, ICGI.

[4]  Pierre Dupont,et al.  Generating annotated behavior models from end-user scenarios , 2005, IEEE Transactions on Software Engineering.

[5]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[6]  D UllmanJeffrey,et al.  Introduction to automata theory, languages, and computation, 2nd edition , 2001 .

[7]  Jeroen Geertzen,et al.  Problems with Evaluation of Unsupervised Empirical Grammatical Inference Systems , 2008, ICGI.

[8]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[9]  Enrique Vidal,et al.  What Is the Search Space of the Regular Inference? , 1994, ICGI.

[10]  Steven P. Reiss,et al.  Encoding program executions , 2001, Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001.

[11]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[12]  Neil Walkinshaw,et al.  Reverse Engineering State Machines by Interactive Grammar Inference , 2007, 14th Working Conference on Reverse Engineering (WCRE 2007).

[13]  Alexander L. Wolf,et al.  Discovering models of software processes from event-based data , 1998, TSEM.

[14]  Menno van Zaanen,et al.  The Omphalos Context-Free Grammar Learning Competition , 2004, ICGI.

[15]  J. Oncina,et al.  INFERRING REGULAR LANGUAGES IN POLYNOMIAL UPDATED TIME , 1992 .

[16]  James R. Larus,et al.  Mining specifications , 2002, POPL '02.

[17]  Geoff Sutcliffe,et al.  The state of CASC , 2006, AI Commun..

[18]  Neil Walkinshaw,et al.  Evaluation and Comparison of Inferred Regular Grammars , 2008, ICGI.

[19]  Colin de la Higuera,et al.  Zulu: An Interactive Learning Competition , 2009, FSMNLP.

[20]  Christos Faloutsos,et al.  Graph evolution: Densification and shrinking diameters , 2006, TKDD.

[21]  Dominique Estival,et al.  The Tenjinno Machine Translation Competition , 2006, ICGI.

[22]  Menno van Zaanen,et al.  Progressing the state-of-the-art in grammatical inference by competition: The Omphalos Context-Free Language Learning Competition , 2005, AI Commun..

[23]  Jerome A. Feldman,et al.  On the Synthesis of Finite-State Machines from Samples of Their Behavior , 1972, IEEE Transactions on Computers.

[24]  Pierre Dupont,et al.  THE QSM ALGORITHM AND ITS APPLICATION TO SOFTWARE BEHAVIOR MODEL INDUCTION , 2008, Appl. Artif. Intell..

[25]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[26]  Alan W. Biermann,et al.  Constructing Programs from Example Computations , 1976, IEEE Transactions on Software Engineering.