Mutation-Based Test Case Generation for Simulink Models

The Matlab/Simulink language has become the standard formalism for modeling and implementing control software in areas like avionics, automotive, railway, and process automation. Such software is often safety critical, and bugs have potentially disastrous consequences for people and material involved. We define a verification methodology to assess the correctness of Simulink programs by means of automated test-case generation. In the style of fault- and mutation-based testing, the coverage of a Simulink program by a test suite is defined in terms of the detection of injected faults. Using bounded model checking techniques, we are able to effectively and automatically compute test suites for given fault models. Several optimisations are discussed to make the approach practical for realistic Simulink programs and fault models, and to obtain accurate coverage measures.

[1]  S. Ramesh,et al.  AutoMOTGen: Automatic Model Oriented Test Generator for Embedded Control Systems , 2008, CAV.

[2]  Niklas Sörensson,et al.  An Extensible SAT-solver , 2003, SAT.

[3]  A Carreno Victor.  Interpretation of IEEE-854 Floating-Point Standard and Definition in the HOL System , 1995 .

[4]  Frank D. Valencia,et al.  Formal Methods for Components and Objects , 2002, Lecture Notes in Computer Science.

[5]  Daniel Kroening,et al.  Mixed abstractions for floating-point arithmetic , 2009, 2009 Formal Methods in Computer-Aided Design.

[6]  Helmut Veith,et al.  FShell: Systematic Test Case Generation for Dynamic Analysis and Measurement , 2008, CAV.

[7]  Gregg Rothermel,et al.  Interactive fault localization techniques in a spreadsheet environment , 2006, IEEE Transactions on Software Engineering.

[8]  Brian Campbell,et al.  Amortised Memory Analysis Using the Depth of Data Structures , 2009, ESOP.

[9]  Antoine Miné,et al.  Relational Abstract Domains for the Detection of Floating-Point Run-Time Errors , 2004, ESOP.

[10]  Ofer Strichman,et al.  Translation Validation: From Simulink to C , 2009, CAV.

[11]  Patrick Cousot,et al.  A static analyzer for large safety-critical software , 2003, PLDI '03.

[12]  Ajitha Rajan,et al.  Requirements Coverage as an Adequacy Measure for Conformance Testing , 2008, ICFEM.

[13]  John Harrison.  Formal Verification of Square Root Algorithms , 2003, Formal Methods Syst. Des..

[14]  Kenneth L. McMillan,et al.  Interpolation and SAT-Based Model Checking , 2003, CAV.

[15]  Sanjit A. Seshia,et al.  A Theory of Mutations with Applications to Vacuity, Coverage, and Fault Tolerance , 2008, 2008 Formal Methods in Computer-Aided Design.

[16]  Stephan Merz,et al.  Model Checking , 2000 .

[17]  Thomas A. Henzinger,et al.  Generating tests from counterexamples , 2004, Proceedings. 26th International Conference on Software Engineering.

[18]  Daniel Kroening,et al.  Behavioral consistency of C and Verilog programs using bounded model checking , 2003, Proceedings 2003. Design Automation Conference (IEEE Cat. No.03CH37451).

[19]  Daniel Kroening,et al.  Coverage in interpolation-based model checking , 2010, Design Automation Conference.

[20]  Mats Per Erik Heimdahl,et al.  Model-Based Safety Analysis of Simulink Models Using SCADE Design Verifier , 2005, SAFECOMP.

[21]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[22]  Hannu Tenhunen,et al.  Extending systems-on-chip to the third dimension: performance, cost and technological tradeoffs , 2007, ICCAD 2007.

[23]  Andreas Zeller,et al.  Efficient mutation testing by checking invariant violations , 2009, ISSTA.

[24]  Mary Sheeran,et al.  Checking Safety Properties Using Induction and a SAT-Solver , 2000, FMCAD.

[25]  Enrico Giunchiglia,et al.  Automatic Test Generation for Coverage Analysis Using CBMC , 2009, EUROCAST.

[26]  Helmut Veith,et al.  Query-Driven Program Testing , 2008, VMCAI.

[27]  Kousha Etessami,et al.  Analysis of Recursive Game Graphs Using Data Flow Equations , 2004, VMCAI.

[28]  David Clark,et al.  Safety and Security Analysis of Object-Oriented Models , 2002, SAFECOMP.

[29]  Bruce H. Krogh,et al.  Hybrid System Verification Is not a Sinecure - the Electronic Throttle Control Case Study , 2006, Int. J. Found. Comput. Sci..

[30]  Thomas Ball,et al.  A Theory of Predicate-Complete Test Coverage and Generation , 2004, FMCO.

[31]  Daniel Kroening,et al.  Checking consistency of C and Verilog using predicate abstraction and induction , 2004, ICCAD 2004.

[32]  Randal E. Bryant,et al.  Graph-Based Algorithms for Boolean Function Manipulation , 1986, IEEE Transactions on Computers.

[33]  A. Kuehlmann,et al.  Combinational and sequential equivalence checking , 2001 .

[34]  Sudeepa Roy,et al.  Tool for Translating Simulink Models into Input Language of a Model Checker , 2006, ICFEM.

[35]  Hassen Saïdi,et al.  Construction of Abstract State Graphs with PVS , 1997, CAV.

[36]  Bruce H. Krogh,et al.  Hybrid System Verification Is Not a Sinecure: The Electronic Throttle Control Case Study , 2004, ATVA.