What am I testing and where? Comparing testing procedures based on lightweight requirements annotations

Context The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop. Objective Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions. Method With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another. Results We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation. Conclusion With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.

[1]  Robert Feldt,et al.  Alignment of Requirements Specification and Testing: A Systematic Mapping Study , 2011, 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.

[2]  Andreas Vogelsang,et al.  Extraction of System States from Natural Language Requirements , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[3]  Marion Berbineau,et al.  Networking Simulation for Intelligent Transportation Systems: High Mobile Wireless Nodes , 2017 .

[4]  J. Banks,et al.  Discrete-Event System Simulation , 1995 .

[5]  Dietmar Pfahl,et al.  Reporting guidelines for controlled experiments in software engineering , 2005, 2005 International Symposium on Empirical Software Engineering, 2005..

[6]  Klaus Beetz,et al.  Model Based Testing of System Requirements using UML Use Case Models , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[7]  Fabiano Dalpiaz.  Requirements data sets (user stories) , 2018 .

[8]  Marjo Kauppinen,et al.  Linking Requirements and Testing in Practice , 2008, 2008 16th IEEE International Requirements Engineering Conference.

[9]  Clémentine Nebut,et al.  Automatic test generation: a use case driven approach , 2006, IEEE Transactions on Software Engineering.

[10]  BrinkkemperSjaak,et al.  Extracting conceptual models from user stories with Visual Narrator , 2017 .

[11]  Tony Gorschek,et al.  Challenges and practices in aligning requirements with verification and validation: a case study of six companies , 2013, Empirical Software Engineering.

[12]  Munindar P. Singh,et al.  Toward Automating Crowd RE , 2017, 2017 IEEE 25th International Requirements Engineering Conference (RE).

[13]  Matthias Jarke,et al.  Toward Reference Models of Requirements Traceability , 2001, IEEE Trans. Software Eng..

[14]  UttingMark,et al.  A taxonomy of model-based testing approaches , 2012 .

[15]  Sjaak Brinkkemper,et al.  Extracting conceptual models from user stories with Visual Narrator , 2017, Requirements Engineering.

[16]  Andreas Vogelsang,et al.  A Lightweight Multilevel Markup Language for Connecting Software Requirements and Simulations , 2019, REFSQ.

[17]  Carl K. Chang,et al.  Event-Based Traceability for Managing Evolutionary Change , 2003, IEEE Trans. Software Eng..

[18]  Alistair Mavin,et al.  Easy Approach to Requirements Syntax (EARS) , 2009, 2009 17th IEEE International Requirements Engineering Conference.

[19]  Andreas Vogelsang,et al.  Challenging Incompleteness of Performance Requirements by Sentence Patterns , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[20]  Olly Gotel,et al.  An analysis of the requirements traceability problem , 1994, Proceedings of IEEE International Conference on Requirements Engineering.

[21]  Thomas Engel,et al.  Luxembourg SUMO Traffic (LuST) Scenario: 24 hours of mobility for vehicular networking research , 2015, 2015 IEEE Vehicular Networking Conference (VNC).

[22]  Xavier Savatier,et al.  Vehicle-Hardware-In-The-Loop system for ADAS prototyping and validation , 2014, 2014 International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation (SAMOS XIV).

[23]  Ajitha Rajan,et al.  Coverage metrics for requirements-based testing , 2006, ISSTA '06.

[24]  Luisa Mich,et al.  Market research for requirements analysis using linguistic tools , 2004, Requirements Engineering.

[25]  WuLing Huang,et al.  Autonomous vehicles testing methods review , 2016, 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC).

[26]  Michael Jackson,et al.  A Reference Model for Requirements and Specifications , 2000, IEEE Softw..

[27]  Colin J. Neill,et al.  State of practice in requirements engineering: contemporary data , 2014, Innovations in Systems and Software Engineering.

[28]  Yun-Pang Flötteröd,et al.  Microscopic Traffic Simulation using SUMO , 2018, 2018 21st International Conference on Intelligent Transportation Systems (ITSC).

[29]  Ilja Radusch,et al.  Simulation of Convergent Networks for Intelligent Transport Systems with VSimRTI , 2017 .

[30]  Hong Zhu,et al.  Software unit test coverage and adequacy , 1997, ACM Comput. Surv..

[31]  Norbert Seyff,et al.  FlexiSketch: a lightweight sketching and metamodeling approach for end-users , 2017, Software & Systems Modeling.