Model Learning and Model-Based Testing

We present a survey of the recent research efforts in integrating model learning with model-based testing. We distinguished two strands of work in this domain, namely test-based learning (also called test-based modeling) and learning-based testing. We classify the results in terms of their underlying models, their test purpose and techniques, and their target domains.

[1]  Christopher Krügel,et al.  Prospex: Protocol Specification Extraction , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[2]  Jan Tretmans,et al.  Approximate Active Learning of Nondeterministic Input Output Transition Systems , 2015, Electron. Commun. Eur. Assoc. Softw. Sci. Technol..

[3]  M. P. Vasilevskii.  Failure diagnosis of automata , 1973 .

[4]  Gusztáv Adamis,et al.  Generating Performance Test Model from Conformance Test Logs , 2015, SDL Forum.

[5]  Ronald L. Rivest,et al.  Inference of finite automata using homing sequences , 1989, STOC '89.

[6]  Shing-Chi Cheung,et al.  Dynamic Model Learning Using Genetic Algorithm under Adaptive Model Checking Framework , 2006, 2006 Sixth International Conference on Quality Software (QSIC'06).

[7]  Gordon Fraser,et al.  Uncertainty-Driven Black-Box Test Data Generation , 2016, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[8]  Takuya Katayama,et al.  Modular Conformance Testing and Assume-Guarantee Verification for Evolving Component-Based Software , 2008, 2008 15th Asia-Pacific Software Engineering Conference.

[9]  Wan Fokkink,et al.  What Can Formal Methods Bring to Systems Biology? , 2009, FM.

[10]  Karl Meinke,et al.  Automated black-box testing of functional correctness using function approximation , 2004, ISSTA '04.

[11]  George C. Necula,et al.  Guided GUI testing of android apps with minimal restart and approximate learning , 2013, OOPSLA.

[12]  Valérie Issarny,et al.  Formal methods for eternal networked software systems : 11th International School on Formal Methods for the Design of Computer, Communication and Software Systems, SFM 2011 ; Bertinoro, Italy, June 13-18, 2011. Advanced lectures , 2011 .

[13]  John Derrick,et al.  Iterative Refinement of Reverse-Engineered Models by Model-Based Testing , 2009, FM.

[14]  Frits W. Vaandrager,et al.  Improving active Mealy machine learning for protocol conformance testing , 2014, Machine Learning.

[15]  Joeri de Ruiter,et al.  Protocol State Fuzzing of TLS Implementations , 2015, USENIX Security Symposium.

[16]  Hasan Sözer,et al.  Automated refinement of models for model-based testing using exploratory testing , 2017, Software Quality Journal.

[17]  David Lee,et al.  A model-based approach to security flaw detection of network protocol implementations , 2008, 2008 IEEE International Conference on Network Protocols.

[18]  Pieter Collins,et al.  Model-Checking in Systems Biology - From Micro to Macro , 2014, FMMB.

[19]  Tiziana Margaria,et al.  Completing and Adapting Models of Biological Processes , 2006, BICC.

[20]  Frits W. Vaandrager,et al.  Learning and Testing the Bounded Retransmission Protocol , 2012, ICGI.

[21]  Roland Groz,et al.  Inferring Mealy Machines , 2009, FM.

[22]  Alex Groce,et al.  Adaptive Model Checking , 2006, Log. J. IGPL.

[23]  Roland Groz,et al.  Analysis and testing of black‐box component‐based systems by inferring partial models , 2014, Softw. Test. Verification Reliab..

[24]  Angelos D. Keromytis,et al.  HVLearn: Automated Black-Box Analysis of Hostname Verification in SSL/TLS Implementations , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[25]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[26]  Simon J. Thompson,et al.  From test cases to FSMs: augmented test-driven development and property inference , 2010, Erlang '10.

[27]  Florentin Ipate,et al.  Model Learning and Test Generation for Event-B Decomposition , 2012, ISoLA.

[28]  Frits W. Vaandrager,et al.  Applying Automata Learning to Embedded Control Software , 2015, ICFEM.

[29]  Keqin Li,et al.  Integration Testing of Distributed Components Based on Learning Parameterized I/O Models , 2006, FORTE.

[30]  Roland Groz,et al.  Automatic Model Inference of Web Applications for Security Testing , 2014, 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.

[31]  Frits W. Vaandrager,et al.  Model learning and model checking of SSH implementations , 2017, SPIN.

[32]  Karl Meinke,et al.  A Learning-Based Approach to Unit Testing of Numerical Software , 2010, ICTSS.

[33]  Paola Inverardi,et al.  Automatic synthesis of behavior protocols for composable web-services , 2009, ESEC/FSE '09.

[34]  Angelos D. Keromytis,et al.  SFADiff: Automated Evasion Attacks and Fingerprinting Using Black-box Differential Automata Learning , 2016, CCS.

[35]  Keqin Li,et al.  Integration Testing of Components Guided by Incremental State Machine Learning , 2006, Testing: Academic & Industrial Conference - Practice And Research Techniques (TAIC PART'06).

[36]  John Derrick,et al.  Increasing Functional Coverage by Inductive Testing: A Case Study , 2010, ICTSS.

[37]  Bernhard Steffen,et al.  Introduction to Active Automata Learning from a Practical Perspective , 2011, SFM.

[38]  M. Reniers,et al.  CHAPTER c 0095 MODEL-BASED TESTING OF CYBER-PHYSICAL SYSTEMS 19 , 2016 .

[39]  Ramon Janssen,et al.  Learning Fragments of the TCP Network Protocol , 2014, FMICS.

[40]  Ina Schaefer,et al.  Towards Efficient and Effective Testing in Automotive Software Development , 2014, GI-Jahrestagung.

[41]  Sheng Yu,et al.  Minimal cover-automata for finite languages , 1998, Theor. Comput. Sci..

[42]  Mohammad Reza Mousavi,et al.  Model-Based Testing of Cyber-Physical Systems , 2017 .

[43]  Jan Tretmans,et al.  Active Learning of Nondeterministic Systems from an ioco Perspective , 2014, ISoLA.

[44]  Bengt Jonsson,et al.  Active learning for extended finite state machines , 2016, Formal Aspects of Computing.

[45]  Ethem Alpaydin,et al.  Introduction to machine learning , 2004, Adaptive computation and machine learning.

[46]  Ferhat Khendek,et al.  Test Selection Based on Finite State Models , 1991, IEEE Trans. Software Eng..

[47]  Tiziana Margaria,et al.  Dynamic testing via automata learning , 2007, International Journal on Software Tools for Technology Transfer.

[48]  Bernhard Steffen,et al.  From ZULU to RERS - Lessons Learned in the ZULU Challenge , 2010, ISoLA.

[49]  David Lee,et al.  Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning , 2008, FORTE.

[50]  Alex Groce,et al.  AMC: An Adaptive Model Checker , 2002, CAV.

[51]  Tiziana Margaria,et al.  Hybrid test of web applications with webtest , 2008, TAV-WEB '08.

[52]  Frits W. Vaandrager,et al.  Learning Register Automata with Fresh Value Generation , 2015, ICTAC.

[53]  G. Myers,et al.  The Art of Software Testing: Myers/Art , 2012 .

[54]  Keqin Li,et al.  Learning Parameterized State Machine Model for Integration Testing , 2007, 31st Annual International Computer Software and Applications Conference (COMPSAC 2007).

[55]  Francesco Bergadano,et al.  Testing by means of inductive program learning , 1996, TSEM.

[56]  Bengt Jonsson,et al.  On the Correspondence Between Conformance Testing and Regular Inference , 2005, FASE.

[57]  Colin de la Higuera,et al.  Zulu: An Interactive Learning Competition , 2009, FSMNLP.

[58]  Bruno Legeard,et al.  A taxonomy of model‐based testing approaches , 2012, Softw. Test. Verification Reliab..

[59]  Edith Elkind,et al.  Grey-Box Checking , 2006, FORTE.

[60]  Petros Papadopoulos,et al.  Black-Box Test Generation from Inferred Models , 2015, 2015 IEEE/ACM 4th International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering.

[61]  Christoph Schulze,et al.  Model generation to support model-based testing applied on the NASA DAT Web-application - An experience report , 2015, 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE).

[62]  Bengt Jonsson,et al.  Learning Extended Finite State Machines , 2014, SEFM.

[63]  Joeri de Ruiter,et al.  Formal Models of Bank Cards for Free , 2013, 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops.

[64]  Dana Angluin,et al.  Learning Regular Sets from Queries and Counterexamples , 1987, Inf. Comput..

[65]  Amer Diwan,et al.  Discovering Documentation for Java Container Classes , 2007, IEEE Transactions on Software Engineering.

[66]  David Lee,et al.  Testing Finite-State Machines: State Identification and Verification , 1994, IEEE Trans. Computers.

[67]  Frits W. Vaandrager,et al.  Inference and Abstraction of the Biometric Passport , 2010, ISoLA.

[68]  Elaine J. Weyuker,et al.  Assessing Test Data Adequacy through Program Inference , 2019, TOPL.

[69]  Florentin Ipate,et al.  Model Learning and Test Generation Using Cover Automata , 2015, Comput. J..

[70]  Wan Fokkink,et al.  Petri Nets Are a Biologist's Best Friend , 2014, FMMB.

[71]  Bernhard K. Aichernig,et al.  Learning from Faults: Mutation Testing in Active Automata Learning , 2017, NFM.

[72]  Jan Tretmans,et al.  Integrating Verification, Testing, and Learning for Cryptographic Protocols , 2007, IFM.

[73]  Mihalis Yannakakis,et al.  Black Box Checking , 1999 .

[74]  Hardi Hungar,et al.  Domain-Specific Optimization in Automata Learning , 2003, CAV.

[75]  Bernhard Steffen,et al.  Automata Learning with Automated Alphabet Abstraction Refinement , 2011, VMCAI.

[76]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[77]  Karl Meinke,et al.  Learning-Based Testing of Distributed Microservice Architectures: Correctness and Fault Injection , 2015, SEFM Workshops.

[78]  Keqin Li,et al.  Modular System Verification by Inference, Testing and Reachability Analysis , 2008, TestCom/FATES.

[79]  Pedro García,et al.  IDENTIFYING REGULAR LANGUAGES IN POLYNOMIAL TIME , 1993 .

[80]  Oliver Niese,et al.  An integrated approach to testing complex systems , 2003 .

[81]  David Lee,et al.  Testing Security Properties of Protocol Implementations - a Machine Learning Based Approach , 2007, 27th International Conference on Distributed Computing Systems (ICDCS '07).

[82]  Hardi Hungar,et al.  Model Generation by Moderated Regular Extrapolation , 2002, FASE.

[83]  Frits W. Vaandrager,et al.  Automata Learning through Counterexample Guided Abstraction Refinement , 2012, FM.

[84]  Mahsa Varshosaz,et al.  Generation of Failure Models through Automata Learning , 2016, 2016 Workshop on Automotive Systems/Software Architectures (WASA).

[85]  Muzammil Shahbaz,et al.  Model Inference Approach for Detecting Feature Interactions in Integrated Systems , 2007, ICFI.

[86]  Karl Meinke,et al.  Incremental Learning-Based Testing for Reactive Systems , 2011, TAP@TOOLS.

[87]  Tiziana Margaria,et al.  Efficient test-based model generation for legacy reactive systems , 2004, Proceedings. Ninth IEEE International High-Level Design Validation and Test Workshop (IEEE Cat. No.04EX940).

[88]  Bernhard K. Aichernig,et al.  Probabilistic Black-Box Reachability Checking , 2017, RV.

[89]  Erik Poll,et al.  Model-Based Testing of Electronic Passports , 2009, FMICS.

[90]  Neil Walkinshaw,et al.  Reverse Engineering State Machines by Interactive Grammar Inference , 2007, 14th Working Conference on Reverse Engineering (WCRE 2007).

[91]  Tsun S. Chow,et al.  Testing Software Design Modeled by Finite-State Machines , 1978, IEEE Transactions on Software Engineering.

[92]  Jean-François Pradat-Peyre,et al.  Formal Techniques for Networked and Distributed Systems - FORTE 2006, 26th IFIP WG 6.1 International Conference, Paris, France, September 26-29, 2006 , 2006, FORTE.

[93]  Fides Aarts,et al.  Tomte : bridging the gap between active learning and real-world systems , 2014 .

[94]  Pierre Dupont,et al.  THE QSM ALGORITHM AND ITS APPLICATION TO SOFTWARE BEHAVIOR MODEL INDUCTION , 2008, Appl. Artif. Intell..

[95]  Bernhard K. Aichernig,et al.  Model-Based Testing IoT Communication via Active Automata Learning , 2017, 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST).

[96]  Frits W. Vaandrager,et al.  Combining Model Learning and Model Checking to Analyze TCP Implementations , 2016, CAV.

[97]  Florentin Ipate,et al.  Learn and Test for Event-B - A Rodin Plugin , 2012, ABZ.

[98]  Umesh V. Vazirani,et al.  An Introduction to Computational Learning Theory , 1994 .

[99]  Jan Tretmans,et al.  Model-Based Testing and Some Steps towards Test-Based Modelling , 2011, SFM.

[100]  Keqin Li,et al.  Inferring Approximated Models for Systems Engineering , 2014, 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering.

[101]  José Oncina,et al.  Learning Stochastic Regular Grammars by Means of a State Merging Method , 1994, ICGI.

[102]  MeinkeKarl.  Automated black-box testing of functional correctness using function approximation , 2004 .

[103]  Dawn Xiaodong Song,et al.  MACE: Model-inference-Assisted Concolic Exploration for Protocol and Vulnerability Discovery , 2011, USENIX Security Symposium.

[104]  Bernhard Steffen,et al.  The Open-Source LearnLib - A Framework for Active Automata Learning , 2015, CAV.

[105]  John Hughes,et al.  Testing telecoms software with quviq QuickCheck , 2006, ERLANG '06.

[106]  Jan Tretmans,et al.  TorX: Automated Model-Based Testing , 2003 .

[107]  Keqin Li,et al.  Learning and Integration of Parameterized Components Through Testing , 2007, TestCom/FATES.