Automated Test Suite Generation for Time-Continuous Simulink Models

All engineering disciplines are founded and rely on models, although they may differ on purposes and usages of modeling. Interdisciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test generation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In contrast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink models. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink.

[1]  Pieter J. Mosterman,et al.  Model-Based Testing for Embedded Systems , 2011, Computational Analysis, Synthesis, & Design Dynamic Systems.

[2]  S. Ramesh,et al.  AutoMOTGen: Automatic Model Oriented Test Generator for Embedded Control Systems , 2008, CAV.

[3]  Daniel Kroening,et al.  Mutation-Based Test Case Generation for Simulink Models , 2009, FMCO.

[4]  Mark Harman,et al.  Coverage and fault detection of the output-uniqueness test selection criteria , 2014, ISSTA 2014.

[5]  Bernhard Rumpe,et al.  Model-driven Development of Complex Software : A Research Roadmap , 2007 .

[6]  S. Ramesh,et al.  Efficient coverage of parallel and hierarchical stateflow models for test case generation , 2012, Softw. Test. Verification Reliab..

[7]  Lubos Brim,et al.  Executing Model Checking Counterexamples in Simulink , 2012, 2012 Sixth International Symposium on Theoretical Aspects of Software Engineering.

[8]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[9]  Stephan Merz,et al.  Model Checking , 2000 .

[10]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[11]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[12]  Lian Duan,et al.  Modeling and requirements on the physical side of cyber-physical systems , 2013, 2013 2nd International Workshop on the Twin Peaks of Requirements and Architecture (TwinPeaks).

[13]  Daniel C. DuVarney,et al.  Experience report: the reactis validation tool , 2007, ICFP '07.

[14]  R. Venkatesh,et al.  Test generation for large automotive models , 2012, 2012 IEEE International Conference on Industrial Technology.

[15]  Mark Harman,et al.  Augmenting test suites effectiveness by increasing output diversity , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[16]  S. Ramesh,et al.  Automatic test case generation from Simulink/Stateflow models using model checking , 2014, Softw. Test. Verification Reliab..

[17]  André Baresel,et al.  Structural and Functional Sequence Test of Dynamic and State-Based Software with Evolutionary Algorithms , 2003, GECCO.

[18]  Mark Harman,et al.  Reducing qualitative human oracle costs associated with automatically generated test data , 2010, STOV '10.

[19]  Thanh Binh Nguyen,et al.  Mutation Operators for Simulink Models , 2012, 2012 Fourth International Conference on Knowledge and Systems Engineering.

[20]  John A. Clark,et al.  Search-based mutation testing for Simulink models , 2005, GECCO '05.

[21]  Rance Cleaveland,et al.  An Instrumentation-Based Approach to Controller Model Validation , 2006, ASWSD.

[22]  Robert Eschbach,et al.  SIMOTEST: A tool for automated testing of hybrid real-time Simulink models , 2011, ETFA2011.

[23]  Justyna Zander-Nowicka,et al.  Model-based Testing of Real-Time Embedded Systems in the Automotive Domain , 2009 .

[24]  Yan Rong Wang,et al.  Research and Improvements on Mutation Operators for Simulink Models , 2014, CIT 2014.

[25]  Alexander Pretschner,et al.  8Cage: lightweight fault-based test generation for simulink , 2014, ASE.

[26]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[27]  Andreas Windisch Search-based test data generation from stateflow statecharts , 2010, GECCO '10.

[28]  A. Sockloff,et al.  Statistical power analysis for the behavioral sciences: (revised edition), by Jacob Cohen. New York: Academic Press, 1977, xv + 474 pp., $24.50. , 1978 .

[29]  Dawson R. Engler,et al.  KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs , 2008, OSDI.

[30]  Alexander Pretschner,et al.  One evaluation of model-based testing and its automation , 2005, ICSE.

[31]  M. Bradac,et al.  Automatic code generation from Matlab/Simulink for critical applications , 2014, 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE).

[32]  Lionel C. Briand,et al.  Effective test suites for mixed discrete-continuous stateflow controllers , 2015, ESEC/SIGSOFT FSE.

[33]  S. Ramesh,et al.  Randomized directed testing (REDIRECT) for Simulink/Stateflow models , 2008, EMSOFT '08.

[34]  Edward A. Lee,et al.  Introduction to Embedded Systems - A Cyber-Physical Systems Approach , 2013 .

[35]  Joachim Wegener,et al.  Integrating Model-Based Testing with Evolutionary Functional Testing , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[36]  Ian Witten,et al.  Data Mining , 2000 .

[37]  References , 1971 .

[38]  Stefan Wappler,et al.  Evolutionary functional testing of continuous control systems , 2009, GECCO.

[39]  Devendra K. Chaturvedi,et al.  Modeling and Simulation of Systems Using MATLAB and Simulink , 2009 .

[40]  Luciano Baresi,et al.  Test oracles for simulink-like models , 2016, Automated Software Engineering.

[41]  Krzysztof Czarnecki,et al.  Generative Programming , 2001, ECOOP Workshops.

[42]  Gordon Fraser,et al.  Evolutionary Generation of Whole Test Suites , 2011, 2011 11th International Conference on Quality Software.

[43]  Gabor Karsai,et al.  Polyglot: modeling and analysis for multiple Statechart formalisms , 2011, ISSTA '11.

[44]  S. Ramesh,et al.  An integrated test generation tool for enhanced coverage of Simulink/Stateflow models , 2012, 2012 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[45]  Emanuele Carpanzano,et al.  Model-Checking based Verification approach for advanced Industrial Automation solutions , 2010, 2010 IEEE 15th Conference on Emerging Technologies & Factory Automation (ETFA 2010).

[46]  Benjamin Wilmes,et al.  Considering Signal Constraints in Search-Based Testing of Continuous Systems , 2010, 2010 Third International Conference on Software Testing, Verification, and Validation Workshops.

[47]  J. Anthony Capon,et al.  Elementary statistics for the social sciences , 1988 .

[48]  Luciano Baresi,et al.  Specifying automated oracles for Simulink models , 2013, 2013 IEEE 19th International Conference on Embedded and Real-Time Computing Systems and Applications.

[49]  Mark Harman,et al.  The Oracle Problem in Software Testing: A Survey , 2015, IEEE Transactions on Software Engineering.

[50]  Andreas Windisch,et al.  Search-based testing of complex simulink models containing stateflow diagrams , 2009, 2009 31st International Conference on Software Engineering - Companion Volume.

[51]  Boaz Porat,et al.  A course in digital signal processing , 1996 .

[52]  John A. Clark,et al.  A search-based framework for automatic testing of MATLAB/Simulink models , 2008, J. Syst. Softw..