Test oracles for simulink-like models

The design of embedded systems is often supported by the definition of executable models for tools like Matlab/Simulink or Scilab/Xcos. These models play a pivotal role in the development process and their correctness is thus extremely important. Many different solutions exist for the definition of suitable tests to “exercise” these models, but only a few (partial) solutions exist for assessing the quality of execution (simulation) results, that is, for defining suitable oracles. This paper addresses the problem and proposes a formal language for specifying the oracles and relating them to existing models. It also presents Apolom, a prototype tool for checking simulation results against stated oracles. The empirical assessment we conducted to assess the viability of the proposed solution is organized around four case studies and witnesses interesting results in terms of effectiveness, efficiency, and required resources.

[1]  Atif M. Memon,et al.  What test oracle should I use for effective GUI testing? , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[2]  Kerry E. Grand,et al.  Large-Scale Modeling for Embedded Applications , 2010 .

[3]  François Delebecque,et al.  Devising Mutant Operators for Dynamic Systems Models by Applying the HAZOP Study , 2011, ICSEA 2011.

[4]  Bernhard K. Aichernig,et al.  Automated Black-Box Testing with Abstract VDM Oracles , 1999, SAFECOMP.

[5]  Aamer Nadeem,et al.  TESTAF: A Test Automation Framework for Class Testing using Object-Oriented Formal Specifications , 2005, J. Univers. Comput. Sci..

[6]  Luciano Baresi,et al.  Keep It Small, Keep It Real: Efficient Run-Time Verification of Web Service Compositions , 2009, FMOODS/FORTE.

[7]  Charles E. McDowell,et al.  Debugging concurrent programs , 1989, ACM Comput. Surv..

[8]  Márcio Eduardo Delamaro,et al.  Test oracles associated with dynamical systems models , 2011 .

[9]  Derrick G. Kourie,et al.  Software testing using model programs , 2001, Softw. Pract. Exp..

[10]  Miguel Felder,et al.  Validating real-time systems by history-checking TRIO specifications , 1992, ICSE '92.

[11]  Sabine Glesner,et al.  Slicing MATLAB Simulink models , 2012, 2012 34th International Conference on Software Engineering (ICSE).

[12]  Rong Chen,et al.  A Method of Log File Analysis for Test Oracle , 2009, 2009 International Conference on Scalable Computing and Communications; Eighth International Conference on Embedded Computing.

[13]  Stephen H. Edwards,et al.  A framework for practical, automated black‐box testing of component‐based software † , 2001, Softw. Test. Verification Reliab..

[14]  Carlo Ghezzi,et al.  TRIO: A logic language for executable specifications of real-time systems , 1990, J. Syst. Softw..

[15]  W. Sweet.  Chernobyl's stressful after-effects , 1996 .

[16]  Robert F. Roggio,et al.  An automated oracle for software testing , 1992 .

[17]  Luciano Baresi,et al.  Specifying automated oracles for Simulink models , 2013, 2013 IEEE 19th International Conference on Embedded and Real-Time Computing Systems and Applications.

[18]  Justyna Zander-Nowicka,et al.  Model-based Testing of Real-Time Embedded Systems in the Automotive Domain , 2009 .

[19]  Marie-Claude Gaudel,et al.  Testing Can Be Formal, Too , 1995, TAPSOFT.

[20]  Hans Vangheluwe,et al.  Instrumentation and preservation of extra-functional properties of simulink models , 2015, SpringSim.

[21]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[22]  Mark Blackburn,et al.  T-VEC: a tool for developing critical systems , 1996, Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96.

[23]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[24]  George S. Avrunin,et al.  Patterns in property specifications for finite-state verification , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[25]  Thorsten Pawletta,et al.  Model-based testing approach for MATLAB/simulink using system entity structure and experimental frames , 2015, SpringSim.

[26]  E. Lorenz.  Deterministic nonperiodic flow , 1963 .

[27]  Xin Wang,et al.  An optimized method for automatic test oracle generation from real-time specification , 2005, 10th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS'05).

[28]  James H. Andrews,et al.  General Test Result Checking with Log File Analysis , 2003, IEEE Trans. Software Eng..

[29]  J. Yorke,et al.  Chaos: An Introduction to Dynamical Systems , 1997 .

[30]  Fernando Santos Osório,et al.  Tiriba - a new approach of UAV based on model driven development and multiprocessors , 2011, 2011 IEEE International Conference on Robotics and Automation.

[31]  Chien-Chang Lee,et al.  Requirements Modeling and Automated Requirements-Based Test Generation , 2013 .

[32]  John V. Guttag,et al.  Abstract data types and the development of data structures , 1977, CACM.

[33]  Magne Haveraaen,et al.  Axiom-Based Transformations: Optimisation and Testing , 2009, Electron. Notes Theor. Comput. Sci..

[34]  John V. Guttag,et al.  Abstract data types and the development of data structures , 1976, Software Pioneers.

[35]  Mark R. Blackburn,et al.  Extending Simulink Models With Natural Relations To Improve Automated Model-Based Testing , 2005, 29th Annual IEEE/NASA Software Engineering Workshop.