An empirical study of injected versus actual interface errors

The reuse of software components is a common practice in commercial applications and increasingly appearing in safety critical systems as driven also by cost considerations. This practice puts dependability at risk, as differing operating conditions in different reuse scenarios may expose residual software faults in the components. Consequently, software fault injection techniques are used to assess how residual faults of reused software components may affect the system, and to identify appropriate counter-measures. As fault injection in components’ code suffers from a number of practical disadvantages, it is often replaced by error injection at the component interface level. However, it is still an open issue, whether such injected errors are actually representative of the effects of residual faults. To this end, we propose a method for analyzing how software faults turn into interface errors, with the ultimate aim of supporting more representative interface error injection experiments. Our analysis in the context of widely used software libraries reveals that existing interface error models are not suitable for emulating software faults, and provides useful insights for improving the representativeness of interface error injection.

[1]  Marco Vieira,et al.  On the emulation of software faults by software fault injection , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[2]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[3]  Peter M. Chen,et al.  The systematic improvement of fault tolerance in the Rio file cache , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[4]  Jacob A. Abraham,et al.  FERRARI: A Flexible Software-Based Fault and Error Injection System , 1995, IEEE Trans. Computers.

[5]  Dong Tang,et al.  An approach to measuring and assessing dependability for critical software systems , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[6]  Nicholas Nethercote,et al.  Valgrind: a framework for heavyweight dynamic binary instrumentation , 2007, PLDI '07.

[7]  Pascale Thévenod-Fosse,et al.  Software error analysis: a real case study involving real faults and mutations , 1996, ISSTA '96.

[8]  Peter M. Chen,et al.  How fail-stop are faulty programs? , 1998, Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224).

[9]  Jeffrey M. Voas,et al.  Predicting How Badly "Good" Software Can Behave , 1997, IEEE Softw..

[10]  Sigrid Eldh,et al.  Component Testing Is Not Enough - A Study of Software Faults in Telecom Middleware , 2007, TestCom/FATES.

[11]  Neeraj Suri,et al.  The impact of fault models on software robustness evaluations , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[12]  Marco Vieira,et al.  A Dependability Benchmark for OLTP Application Environments , 2003, VLDB.

[13]  Neeraj Suri,et al.  EPIC: profiling the propagation and effect of data errors in software , 2004, IEEE Transactions on Computers.

[14]  Thomas G. Szymanski,et al.  A fast algorithm for computing longest common subsequences , 1977, CACM.

[15]  James Cownie,et al.  PinPlay: a framework for deterministic replay and reproducible analysis of parallel programs , 2010, CGO '10.

[16]  Nuno Laranjeiro,et al.  A Technique for Deploying Robust Web Services , 2014, IEEE Transactions on Services Computing.

[17]  Henrique Madeira,et al.  Emulation of Software Faults: A Field Data Study and a Practical Approach , 2006, IEEE Transactions on Software Engineering.

[18]  Eliane Martins,et al.  Experimental Risk Assessment and Comparison Using Software Fault Injection , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).

[19]  Domenico Cotroneo,et al.  Fault Injection for Software Certification , 2013, IEEE Security & Privacy.

[20]  Xiangyu Zhang,et al.  Enabling tracing Of long-running multithreaded programs via dynamic execution reduction , 2007, ISSTA '07.

[21]  Ravishankar K. Iyer,et al.  FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults , 1993, IEEE Trans. Software Eng..

[22]  Domenico Cotroneo,et al.  Experimental Analysis of Binary-Level Software Fault Injection in Complex Software , 2012, 2012 Ninth European Dependable Computing Conference.

[23]  Elaine J. Weyuker,et al.  Testing Component-Based Software: A Cautionary Tale , 1998, IEEE Softw..

[24]  Peter M. Chen,et al.  The Design and Verification of the Rio File Cache , 2001, IEEE Trans. Computers.

[25]  Jean Arlat,et al.  Analysis of the effects of real and injected software faults: Linux as a case study , 2002, 2002 Pacific Rim International Symposium on Dependable Computing, 2002. Proceedings..

[26]  Jean Arlat,et al.  Dependability of COTS Microkernel-Based Systems , 2002, IEEE Trans. Computers.

[27]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[28]  Ram Chillarege,et al.  Generation of an error set that emulates software faults based on field data , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[29]  Maliha S. Nash,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 2001, Technometrics.

[30]  Karama Kanoun,et al.  Dependability benchmarking for computer systems , 2008 .

[31]  J.H. Andrews,et al.  Is mutation an appropriate tool for testing experiments? [software testing] , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[32]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[33]  Gregg Rothermel,et al.  On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques , 2006, IEEE Transactions on Software Engineering.

[34]  Domenico Cotroneo,et al.  On Fault Representativeness of Software Fault Injection , 2013, IEEE Transactions on Software Engineering.

[35]  Anup K. Ghosh,et al.  An approach to testing COTS software for robustness to operating system exceptions and errors , 1999, Proceedings 10th International Symposium on Software Reliability Engineering (Cat. No.PR00443).

[36]  Daniel P. Siewiorek,et al.  Fault Injection Experiments Using FIAT , 1990, IEEE Trans. Computers.

[37]  George Candea,et al.  Efficient Testing of Recovery Code Using Fault Injection , 2011, TOCS.

[38]  Jeffrey M. Voas,et al.  Certifying Off-the-Shelf Software Components , 1998, Computer.

[39]  Philip Koopman,et al.  The Exception Handling Effectiveness of POSIX Operating Systems , 2000, IEEE Trans. Software Eng..

[40]  Mark Harman,et al.  An Analysis and Survey of the Development of Mutation Testing , 2011, IEEE Transactions on Software Engineering.

[41]  Martin Hiller,et al.  An experimental comparison of fault and error injection , 1998, Proceedings Ninth International Symposium on Software Reliability Engineering (Cat. No.98TB100257).

[42]  Eliane Martins,et al.  Injection of faults at component interfaces and inside the component code: are they equivalent? , 2006, 2006 Sixth European Dependable Computing Conference.