An empirical study on the application of mutation testing for a safety-critical industrial software system

Background: Testing is an essential activity in safety-critical software development, following high standards in terms of code coverage. Mutation testing allows assessing the effectiveness of testing and helps to further improve test cases. However, mutation testing is not widely practiced due to scalability problems when applied to real-world systems. Objective: The objective of the study is to investigate the applicability and usefulness of mutation testing for improving the quality of unit testing in context of safety-critical software systems. Method: A case study has been conducted together with an engineering company developing safety-critical systems. Mutation analysis has been applied to the studied system under test (60,000 LOC of C code) producing 75,043 mutants of which 27,158 survived test execution. A sample of 200 live mutants has been reviewed by the engineers, who also improved the existing unit test suite based on their findings. Findings: The reviewed sample contained 24+ equivalent mutants and 12+ duplicated mutants. It revealed a weak spot in the testing approach and provided valuable guidance to improve the existing unit test suite. Two new faults were found in the code when improving the tests. Test execution against the mutants required over 4,000 hours computing time. The overall effort was about half a person year.