Evaluating Specification-level MC/DC Criterion in Model-Based Testing of Safety Critical Systems

Safety-critical software systems in the aviation domain, e.g., a UAV autopilot software, needs to go through a formal process of certification (e.g., DO-178C standard). One of the main requirements for this certification is having a set of explicit test cases for each software requirement. To achieve this, the DO-178C standard recommends using a model-driven approach. For instance, model-based testing (MBT) is recommended in its DO-331 supplement to automatically generate system-level test cases for the requirements provided as the specification models. In addition, the DO-178C standard also requires high level of source code coverage, which typically is achieved by a separate set of structural testing. However, the standard allows targeting high code coverage with MBT, only if the applicants justify their plan on how to achieve high code coverage through model-level testing. In this study, we propose using the Modified Condition and Decision coverage ("MC/DC") criterion on the specification-level constraints rather than the standard-recommended "all transition coverage" criterion, to achieve higher code coverage through MBT. We evaluate our approach in the context of a case study at MicroPilot Inc., our industry collaborator, which is a UAV producer company. We implemented our idea as an MC/DC coverage on transition guards in a UML state-machine-based testing tool that was developed in-house. The results show that applying model-level MC/DC coverage outperforms the typical transition-coverage (DO-178C's required MBT coverage criterion), with respect to source code-level "all condition-decision coverage criterion" by 33%. In addition, our MC/DC test suite detected three new faults and two instances of legacy specification in the code that are no longer in use, compared to the "all transition" test suite.

[1]  Günther Ruhe,et al.  Search Based Software Engineering , 2013, Lecture Notes in Computer Science.

[2]  David Binkley,et al.  Unravel:: a case tool to assist evaluation of high integrity software , 1995 .

[3]  George C. Necula,et al.  CIL: Intermediate Language and Tools for Analysis and Transformation of C Programs , 2002, CC.

[4]  Lionel C. Briand,et al.  Achieving scalable model-based testing through test case diversity , 2013, TSEM.

[5]  Dolores R. Wallace,et al.  Using the Unravel Program Slicing Tool to Evaluate High Integrity Software , 1997 .

[6]  Joachim Wegener,et al.  Evolutionary test environment for automatic structural testing , 2001, Inf. Softw. Technol..

[7]  Nikolai Kosmatov,et al.  Frama-C - A Software Analysis Perspective , 2012, SEFM.

[8]  Gordon Fraser,et al.  An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP).

[9]  Mohammad Reza Mousavi,et al.  Integrating Model-Based and Constraint-Based Testing Using SpecExplorer , 2012, 2012 IEEE 23rd International Symposium on Software Reliability Engineering Workshops.

[10]  Nikolai Tillmann,et al.  Pex-White Box Test Generation for .NET , 2008, TAP.

[11]  Nigel James Tracey,et al.  A search-based automated test-data generation framework for safety-critical software , 2000 .

[12]  Lionel C. Briand,et al.  Generating Test Data from OCL Constraints with Search Techniques , 2013, IEEE Transactions on Software Engineering.

[13]  Nikolai Tillmann,et al.  Achieving both model and code coverage with automated gray-box testing , 2007, A-MOST '07.

[14]  Nikolai Kosmatov,et al.  Frama-C: A software analysis perspective , 2015, Formal Aspects of Computing.

[15]  Mehrdad Sabetzadeh,et al.  An extended systematic literature review on provision of evidence for safety certification , 2014, Inf. Softw. Technol..

[16]  Nikolaj Bjørner,et al.  Z3: An Efficient SMT Solver , 2008, TACAS.

[17]  David W. Binkley,et al.  Interprocedural slicing using dependence graphs , 1990, TOPL.

[18]  Hoyt Lougee,et al.  SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION , 2001 .

[19]  G. S. Prashanth,et al.  Increase in Modified Condition/Decision Coverage using program code transformer , 2013, 2013 3rd IEEE International Advance Computing Conference (IACC).

[20]  David W. Binkley,et al.  Program slicing , 2008, 2008 Frontiers of Software Maintenance.

[21]  Raghu Kacker,et al.  Improving MC/DC and Fault Detection Strength Using Combinatorial Testing , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C).

[22]  Gordon Fraser,et al.  Whole Test Suite Generation , 2013, IEEE Transactions on Software Engineering.

[23]  Koushik Sen,et al.  Heuristics for Scalable Dynamic Test Generation , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[24]  Lionel C. Briand,et al.  A Systematic Review of the Application and Empirical Investigation of Search-Based Test Case Generation , 2010, IEEE Transactions on Software Engineering.