Electronic Notes in Theoretical Computer Science

AbstractEmbeddedsystemsareofgrowingimportanceinindustry. Forexample,inatoday'svehicleahugenumberofembeddedandcommunicating systems can be found. Exhaustive testing of such systems is a requirement, because changes after deliveryand use are expensive and sometimes even impossible. In this paper we propose the use of qualitative models, which are anabstraction of quantitative physical models, for test case generation and test execution. In particular, we show how Simulinkmodels from which control programs are automatically extracted can be tested with respect to qualitative models. SinceSimulink models are heavily used in industry, the approach is of practical interest.Keywords: conformance testing, hybrid systems, qualitative reasoning, qrioconf, Garp3 1 Introduction In industry and especially in the automotive industry Simulink is often used to implementcontrol programs for various purposes. One reason is that those models can be directlyconverted into C code, which runs on the vehicle's electronic control units (ECUs). As aconsequence Simulink models have to be tested thoroughly. This holds in particular forsafety critical systems. In order to meet the safety and quality criteria of such modelsautomated test case generation and more specically model-based testing is of specicinterest but has hardly been explored. In order to ll this gap we present an approach thatmakes use of qualitative models for model-based test cases generation.Qualitative models represent basically cause-effect relationships and constraints onmodel variables. They can be seen as an abstraction of the usually implemented quan-titative differential equation models when using Simulink or other modeling languages.Hence, Simulink models are a renement of qualitative models. This is in contrast to theuse of other means for representing models in this domain like hybrid automata, whichshares basically the same abstraction level with Simulink models.Inordertoallowforusingqualitativemodelsformodel-basedtestingwehavetospecifythe equality relation between the specication and the implementation. For this purpose

[1]  Bert Bredeweg,et al.  Curriculum for learning about QR modelling , Naturnet-Redime, STREP project co-funded by the European Commission within the Sixth Framework Programme (2002-2006), Project no. 004074, Project Deliverable Report D6.9.1. , 2006 .

[2]  Jayadev Misra A discipline of multiprogramming , 1996, CSUR.

[3]  Junfeng Yang,et al.  Using model checking to find serious file system errors , 2004, TOCS.

[4]  Ralph-Johan Back,et al.  Distributed cooperation with action systems , 1988, TOPL.

[5]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[6]  Duncan Clarke,et al.  STG: a tool for generating symbolic test programs and oracles from operational specifications , 2001, ESEC/FSE-9.

[7]  Peter Struss,et al.  Testing Physical Systems , 1994, AAAI.

[8]  Patrick Cousot,et al.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints , 1977, POPL.

[9]  Leonidas Tsiopoulos,et al.  Formal Development of NoC Systems in B , 2006, Nord. J. Comput..

[10]  Pierre Marquis,et al.  A Knowledge Compilation Map , 2002, J. Artif. Intell. Res..

[11]  Orna Grumberg,et al.  Bounded Model Checking of Concurrent Programs , 2005, CAV.

[12]  Dawson R. Engler,et al.  RacerX: effective, static detection of race conditions and deadlocks , 2003, SOSP '03.

[13]  C. Csallner,et al.  Check 'n' crash: combining static checking and testing , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[14]  George S. Avrunin,et al.  Patterns in property specifications for finite-state verification , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[15]  Fabrice Bouquet,et al.  Symbolic Animation of JML Specifications , 2005, FM.

[16]  K. Rustan M. Leino,et al.  Practical Reasoning About Invocations and Implementations of Pure Methods , 2007, FASE.

[17]  Bertrand Meyer,et al.  Automatic Testing of Object-Oriented Software , 2007, SOFSEM.

[18]  Richard G. Hamlet,et al.  Partition Testing Does Not Inspire Confidence , 1990, IEEE Trans. Software Eng..

[19]  Bert Bredeweg,et al.  User Manual for Collaborative QR model building and simulation workbench, Naturnet-Redime, STREP project co-funded by the European Commission within the Sixth Framework Programme (2002-2006), Project no. 004074, Project Deliverable Report D4.2.2 , 2006 .

[20]  Jean-Louis Lanet,et al.  Formal Proof of Smart Card Applets Correctness , 1998, CARDIS.

[21]  Lydie du Bousquet,et al.  Filtering TOBIAS Combinatorial Test Suites , 2004, FASE.

[22]  Bernhard K. Aichernig,et al.  Test case generation by OCL mutation and constraint solving , 2005, Fifth International Conference on Quality Software (QSIC'05).

[23]  Jan Tretmans,et al.  Conformance Testing with Labelled Transition Systems: Implementation Relations and Test Generation , 1996, Comput. Networks ISDN Syst..

[24]  María-del-Mar Gallardo,et al.  Model Extraction for ARINC 653 Based Avionics Software , 2007, SPIN.

[25]  Michael J. Butler,et al.  ProTest: An Automatic Test Environment for B Specifications , 2004, MBT.

[26]  Edmund M. Clarke,et al.  Model Checking , 1999, Handbook of Automated Reasoning.

[27]  Yuri Gurevich,et al.  Sequential abstract-state machines capture sequential algorithms , 2000, TOCL.

[28]  Ecma,et al.  Common Language Infrastructure (CLI) , 2001 .

[29]  Bernhard K. Aichernig,et al.  Mutation testing in UTP , 2009, Formal Aspects of Computing.

[30]  Nikolaj Bjørner,et al.  Using Dynamic Symbolic Execution to Improve Deductive Verification , 2008, SPIN.

[31]  Frédéric Dadeau,et al.  Combining Scenario- and Model-Based Testing to Ensure POSIX Compliance , 2008, ABZ.

[32]  Krishna R. Pattipati,et al.  An Integrated Diagnostic Development Process for Automotive Engine Control Systems , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[33]  Fabrice Bouquet,et al.  CLPS–B – A constraint solver to animate a B specification , 2004, International Journal on Software Tools for Technology Transfer.

[34]  Gordon Fraser,et al.  Qr-model based testing , 2008, AST '08.

[35]  Paul Benoit,et al.  Météor: A Successful Application of B in a Large Project , 1999, World Congress on Formal Methods.

[36]  Harold W. Thimbleby,et al.  The directed Chinese Postman Problem , 2003, Softw. Pract. Exp..

[37]  Michiel van Osch Hybrid Input-Output Conformance and Test Generation , 2006, FATES/RV.

[38]  Boris Beizer,et al.  Black Box Testing: Techniques for Functional Testing of Software and Systems , 1996, IEEE Software.

[39]  Koushik Sen,et al.  DART: directed automated random testing , 2005, PLDI '05.

[40]  Dinghao Wu,et al.  KISS: keep it simple and sequential , 2004, PLDI '04.

[41]  Stefan Heinz,et al.  Using Model Counting to Find Optimal Distinguishing Tests , 2009, CPAIOR.

[42]  Rajeev Alur,et al.  Distinguishing tests for nondeterministic and probabilistic machines , 1995, STOC '95.

[43]  Jacques Julliand,et al.  Generating security tests in addition to functional tests , 2008, AST '08.

[44]  T. Henzinger The theory of hybrid automata , 1996, LICS 1996.

[45]  Ivar Jacobson,et al.  The unified modeling language reference manual , 2010 .

[46]  Ralph-Johan Back,et al.  Decentralization of Process Nets with Centralized Control , 1983, PODC.

[47]  Shaz Qadeer,et al.  CHESS: A Systematic Testing Tool for Concurrent Software , 2007 .

[48]  Alexandre Petrenko,et al.  Can a Model Checker Generate Tests for Non-Deterministic Systems? , 2007, MBT.

[49]  Floris Linnebank,et al.  Garp3: a new workbench for qualitative reasoning and modelling , 2007, K-CAP '07.

[50]  Jacques Julliand,et al.  Generating Tests from B Specifications and Test Purposes , 2008, ABZ.

[51]  Thierry Jéron,et al.  TGV : theory , principles and algorithms A tool for the automatic synthesis of conformance test cases for non-deterministic reactive systems , 2004 .

[52]  Dawson R. Engler,et al.  EXE: automatically generating inputs of death , 2006, CCS '06.

[53]  Patrice Godefroid,et al.  Software Model Checking: The VeriSoft Approach , 2005, Formal Methods Syst. Des..

[54]  Gerard J. Holzmann,et al.  Model-Driven Software Verification , 2004, SPIN.

[55]  Gerard J. Holzmann,et al.  The SPIN Model Checker , 2003 .

[56]  Edsger W. Dijkstra,et al.  Predicate Calculus and Program Semantics , 1989, Texts and Monographs in Computer Science.

[57]  Frank Waters,et al.  The B Book , 1971 .

[58]  Frank Maurer,et al.  SCENTOR: scenario-based testing of e-business applications , 2001, Proceedings Tenth IEEE International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. WET ICE 2001.

[59]  Julien Schmaltz,et al.  A Functional Approach to the Formal Specification of Networks on Chip , 2004, FMCAD.

[60]  Ying Jiang,et al.  Contract-based mutation for testing components , 2005, 21st IEEE International Conference on Software Maintenance (ICSM'05).

[61]  Jeremy Dick,et al.  Automating the Generation and Sequencing of Test Cases from Model-Based Specifications , 1993, FME.

[62]  Koushik Sen,et al.  CUTE: a concolic unit testing engine for C , 2005, ESEC/FSE-13.

[63]  Robert V. Binder,et al.  Testing Object-Oriented Systems: Models, Patterns, and Tools , 1999 .

[64]  Alain Denise,et al.  Coverage-biased Random Exploration of Models , 2008, Electron. Notes Theor. Comput. Sci..

[65]  Bruno Legeard,et al.  LEIRIOS Test Generator: Automated Test Generation from B Models , 2007, B.

[66]  Catherine Oriat,et al.  Jartege: A Tool for Random Generation of Unit Tests for Java Classes , 2004, QoSA/SOQUA.

[67]  Bruno Legeard,et al.  Generation of test sequences from formal specifications: GSM 11‐11 standard case study , 2004, Softw. Pract. Exp..

[68]  Insup Lee,et al.  Robust Test Generation and Coverage for Hybrid Systems , 2007, HSCC.

[69]  Michael Burrows,et al.  Eraser: a dynamic data race detector for multithreaded programs , 1997, TOCS.

[70]  Michael Butler,et al.  Distributed System Development in B , 1996 .

[71]  Margus Veanes,et al.  Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer , 2008, Formal Methods and Testing.

[72]  Alex Groce,et al.  Random testing and model checking: building a common framework for nondeterministic exploration , 2008, WODA '08.

[73]  William J. Dally,et al.  Route packets, not wires: on-chip inteconnection networks , 2001, DAC '01.

[74]  David Lee,et al.  Principles and methods of testing finite state machines-a survey , 1996, Proc. IEEE.

[75]  Raymond A. Paul,et al.  Scenario-based object-oriented testing framework , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[76]  Shaoying Liu,et al.  Criteria for generating specification-based tests , 1999, Proceedings Fifth IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'99) (Cat. No.PR00434).

[77]  Carroll Morgan,et al.  Programming from specifications , 1990, Prentice Hall International Series in computer science.

[78]  Dawson R. Engler,et al.  Checking system rules using system-specific, programmer-written compiler extensions , 2000, OSDI.

[79]  Koushik Sen,et al.  Effective random testing of concurrent programs , 2007, ASE.

[80]  Kaisa Sere,et al.  Reasoning about Action Systems using the B-Method , 1998, Formal Methods Syst. Des..

[81]  Pietro Ferrara,et al.  Safer unsafe code for .NET , 2008, OOPSLA '08.

[82]  K. Rustan M. Leino,et al.  Weakest-precondition of unstructured programs , 2005, PASTE '05.

[83]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[84]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[85]  Dawson R. Engler,et al.  Under-constrained execution: making automatic code destruction easy and scalable , 2007, ISSTA '07.

[86]  Kaisa Sere,et al.  Asynchronous system synthesis , 2005, Sci. Comput. Program..

[87]  Harlan D. Mills,et al.  Theory of Modules , 1987, IEEE Transactions on Software Engineering.

[88]  Philippe Flajolet,et al.  A Calculus for the Random Generation of Labelled Combinatorial Structures , 1994, Theor. Comput. Sci..

[89]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[90]  Kaisa Sere,et al.  Hybrid action systems , 2003, Theor. Comput. Sci..

[91]  Leonardo de Moura,et al.  Automated Test Generation with SAL , 2005 .

[92]  Timothy J. Hickey,et al.  Rigorous Modeling of Hybrid Systems Using Interval Arithmetic Constraints , 2004, HSCC.

[93]  Benjamin Kuipers,et al.  Qualitative reasoning: Modeling and simulation with incomplete knowledge , 1994, Autom..

[94]  Frédérique Bassino,et al.  Random generation of possibly incomplete deterministic automata. , 2008 .

[95]  J. Esposito Randomized test case generation for hybrid systems: metric selection , 2004, Thirty-Sixth Southeastern Symposium on System Theory, 2004. Proceedings of the.

[96]  Jinbo Huang,et al.  Combining Knowledge Compilation and Search for Conformant Probabilistic Planning , 2006, ICAPS.

[97]  Gordon Fraser,et al.  Coverage-based Testing Using Qualitative Reasoning Models , 2008, SEKE.

[98]  Marco Benedetti,et al.  sKizzo: A Suite to Evaluate and Certify QBFs , 2005, CADE.

[99]  Benjamin J. Kaipers,et al.  Qualitative Simulation , 1989, Artif. Intell..

[100]  Thomas A. Henzinger,et al.  Race checking by context inference , 2004, PLDI '04.

[101]  Bor-Yuh Evan Chang,et al.  Boogie: A Modular Reusable Verifier for Object-Oriented Programs , 2005, FMCO.

[102]  Frédérique Bassino,et al.  : A Library to Randomly and Exhaustively Generate Automata , 2007, CIAA.

[103]  Matthew B. Dwyer,et al.  Parallel Randomized State-Space Search , 2007, 29th International Conference on Software Engineering (ICSE'07).

[104]  Cliff B. Jones,et al.  Systematic software development using VDM , 1986, Prentice Hall International Series in Computer Science.

[105]  Thomas A. Henzinger,et al.  HYTECH: a model checker for hybrid systems , 1997, International Journal on Software Tools for Technology Transfer.

[106]  Michael D. Ernst,et al.  Feedback-Directed Random Test Generation , 2007, 29th International Conference on Software Engineering (ICSE'07).

[107]  Peter Struss,et al.  Fault-Model-Based Test Generation for Embedded Software , 2007, IJCAI.

[108]  Tarik Nahhal,et al.  Test Coverage for Continuous and Hybrid Systems , 2007, CAV.

[109]  Frédérique Bassino,et al.  Enumeration and random generation of accessible automata , 2007, Theor. Comput. Sci..

[110]  R. Tourki,et al.  Transaction level modeling of an OSI-like layered NoC , 2006, International Conference on Design and Test of Integrated Systems in Nanoscale Technology, 2006. DTIS 2006..

[111]  Kenneth D. Forbus Qualitative Process Theory , 1984, Artif. Intell..

[112]  James Bret Michael,et al.  Environment behavior models for scenario generation and testing automation , 2005, ACM SIGSOFT Softw. Eng. Notes.

[113]  Tobias Bjerregaard,et al.  A survey of research and practices of Network-on-chip , 2006, CSUR.

[114]  Michael J. Butler,et al.  ProB: A Model Checker for B , 2003, FME.

[115]  Simeon C. Ntafos,et al.  A report on random testing , 1981, ICSE '81.