Xception fault injection and robustness testing framework : a case-study of testing RTEMS

Xception is an automated and comprehensive fault injection and robustness testing environment that enables accurate and flexible V&V (verification & validation) and evaluation of mission and business critical computer systems and computer components, with particular emphasis to software components. In this paper we focus on the new robustness testing features of Xception and illustrate them with a concrete example of robustness testing of the Real Time Executive for Multiprocessor Systems (RTEMS) performed under a European Space Agency (ESA) contract. To the best of our knowledge, this is the first time that robustness testing results for this real time operating system are presented. The testing revealed a significant number of critical flaws in RTEMS 4.5.0 and shows the effectiveness of Xception toolset.

[1]  Henrique Madeira,et al.  Xception: Software Fault Injection and Monitoring in Processor Functional Units1 , 1995 .

[2]  Ravishankar K. Iyer,et al.  An approach towards benchmarking of fault-tolerant commercial systems , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[3]  Daniel P. Siewiorek,et al.  Comparing operating systems using robustness benchmarks , 1997, Proceedings of SRDS'97: 16th IEEE Symposium on Reliable Distributed Systems.

[4]  Gary McGraw,et al.  Software fault injection: inoculating programs against errors , 1997 .

[5]  Henrique Madeira,et al.  Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers , 1998, IEEE Trans. Software Eng..

[6]  Daniel P. Siewiorek,et al.  Automated robustness testing of off-the-shelf software components , 1998, Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224).

[7]  Philip Koopman,et al.  Comparing the robustness of POSIX operating systems , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[8]  Jean Arlat,et al.  MAFALDA: Microkernel Assessment by Fault Injection and Design Aid , 1999, EDCC.

[9]  Henrique Madeira,et al.  INJECTION OF FAULTS IN COMPLEX COMPUTERS , 2001 .

[10]  Jean Arlat,et al.  Dependability of COTS Microkernel-Based Systems , 2002, IEEE Trans. Computers.

[11]  J. Karlsson,et al.  GOOFI: generic object-oriented fault injection tool , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[12]  Eliane Martins,et al.  Architecture-based Strategy for Interface Fault Injection , 2004 .