Experimental Analysis of Binary-Level Software Fault Injection in Complex Software

The injection of software faults (i.e., bugs) by mutating the binary executable code of a program enables the experimental dependability evaluation of systems for which the source code is not available. This approach requires that programming constructs used in the source code should be identified by looking only at the binary code, since the injection is performed at this level. Unfortunately, it is a difficult task to inject faults in the binary code that correctly emulate software defects in the source code. The accuracy of binary-level software fault injection techniques is therefore a major concern for their adoption in real-world scenarios. In this work, we propose a method for assessing the accuracy of binary-level fault injection, and provide an extensive experimental evaluation of a binary-level technique, G-SWFIT, in order to assess its limitations in a real-world complex software system. We injected more than 12 thousand binary-level faults in the OS and application code of the system, and we compared them with faults injected in the source code by using the same fault types of G-SWFIT. The method was effective at highlighting the pitfalls that can occur in the implementation of G-SWFIT. Our analysis shows that G-SWFIT can achieve an improved degree of accuracy if these pitfalls are avoided.

[1]  Tânia Basso,et al.  J-SWFIT: A Java Software Fault Injection Tool , 2011, 2011 5th Latin-American Symposium on Dependable Computing.

[2]  Henrique Madeira,et al.  Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers , 1998, IEEE Trans. Software Eng..

[3]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[4]  Archana Ganapathi,et al.  Why Do Internet Services Fail, and What Can Be Done About It? , 2002, USENIX Symposium on Internet Technologies and Systems.

[5]  Ram Chillarege,et al.  Generation of an error set that emulates software faults based on field data , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[6]  Ravishankar K. Iyer,et al.  NFTAPE: a framework for assessing dependability in distributed systems with lightweight fault injectors , 2000, Proceedings IEEE International Computer Performance and Dependability Symposium. IPDS 2000.

[7]  Karama Kanoun,et al.  DeBERT: Dependability Benchmarking of Embedded Real-Time Off-the-Shelf Components for Space Applications , 2008 .

[8]  Marco Vieira,et al.  On the emulation of software faults by software fault injection , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[9]  Karama Kanoun,et al.  Dependability benchmarking for computer systems , 2008 .

[10]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[11]  Jean Arlat,et al.  Analysis of the effects of real and injected software faults: Linux as a case study , 2002, 2002 Pacific Rim International Symposium on Dependable Computing, 2002. Proceedings..

[12]  Cecília M. F. Rubira,et al.  Jaca: a reflective fault injection tool based on patterns , 2002, Proceedings International Conference on Dependable Systems and Networks.

[13]  Ravishankar K. Iyer,et al.  DEFINE: a distributed fault injection and monitoring environment , 1994, Proceedings of IEEE Workshop on Fault-Tolerant Parallel and Distributed Systems.

[14]  Steven K. Thompson,et al.  Sample Size for Estimating Multinomial Proportions , 1987 .

[15]  Jianhui Jiang,et al.  Fault Injection Scheme for Embedded Systems at Machine Code Level and Verification , 2009, 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing.

[16]  Mark Sullivan,et al.  Software defects and their impact on system availability-a study of field failures in operating systems , 1991, [1991] Digest of Papers. Fault-Tolerant Computing: The Twenty-First International Symposium.

[17]  Henrique Madeira,et al.  Generic faultloads based on software faults for dependability benchmarking , 2004, International Conference on Dependable Systems and Networks, 2004.

[18]  Jean Arlat,et al.  Fault Injection for Dependability Validation: A Methodology and Some Applications , 1990, IEEE Trans. Software Eng..

[19]  Peter M. Chen,et al.  The systematic improvement of fault tolerance in the Rio file cache , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[20]  Daniel P. Siewiorek,et al.  Fault Injection Experiments Using FIAT , 1990, IEEE Trans. Computers.

[21]  Kai Qian,et al.  Embedded Software Development with C , 2009 .

[22]  Jacob A. Abraham,et al.  FERRARI: A Flexible Software-Based Fault and Error Injection System , 1995, IEEE Trans. Computers.

[23]  Jim Gray,et al.  A census of Tandem system availability between 1985 and 1990 , 1990 .

[24]  Philip Koopman,et al.  The Exception Handling Effectiveness of POSIX Operating Systems , 2000, IEEE Trans. Software Eng..

[25]  Ravishankar K. Iyer,et al.  FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults , 1993, IEEE Trans. Software Eng..

[26]  Johan Karlsson,et al.  GOOFI: generic object-oriented fault injection tool , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[27]  Henrique Madeira,et al.  Emulation of software faults by educated mutations at machine-code level , 2002, 13th International Symposium on Software Reliability Engineering, 2002. Proceedings..

[28]  Henrique Madeira,et al.  Emulation of Software Faults: A Field Data Study and a Practical Approach , 2006, IEEE Transactions on Software Engineering.

[29]  Eliane Martins,et al.  Experimental Risk Assessment and Comparison Using Software Fault Injection , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).

[30]  D. P. Siewiorek,et al.  Evaluation and comparison of fault-tolerant software techniques , 1993 .

[31]  Lorenzo Strigini,et al.  Assessing Asymmetric Fault-Tolerant Software , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[32]  Jeffrey M. Voas,et al.  Predicting How Badly "Good" Software Can Behave , 1997, IEEE Softw..

[33]  Jean Arlat,et al.  Dependability of COTS Microkernel-Based Systems , 2002, IEEE Trans. Computers.

[34]  Anup K. Ghosh,et al.  Testing the robustness of Windows NT software , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[35]  Eliane Martins,et al.  Injection of faults at component interfaces and inside the component code: are they equivalent? , 2006, 2006 Sixth European Dependable Computing Conference.