Windows and Linux Robustness Benchmarks with Respect to Application Erroneous Behavior

This chapter presents the specification of dependability benchmarks for general-purpose operating systems with respect to application erroneous behavior, and shows examples of benchmark results obtained for various versions of Windows and Linux operating systems. The benchmark measures are: operating system robustness (as regards possible erroneous inputs provided by the application software to the operating system (OS) via the application programming interface), the OS reaction, and restart times in the presence of faults. Two workloads are used for implementing the benchmark: PostMark, a file system performance benchmark for operating systems, and the Java Virtual Machine (JVM) middleware, a software layer on top of the OS allowing applications in Java language to be platform independent.

[1]  Jean Arlat,et al.  Dependability of COTS Microkernel-Based Systems , 2002, IEEE Trans. Computers.

[2]  Yves Crouzet,et al.  Benchmarking the dependability of Windows and Linux using PostMark/spl trade/ workloads , 2005, 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05).

[3]  Henrique Madeira,et al.  Characterization of operating systems behavior in the presence of faulty drivers through software fault emulation , 2002, 2002 Pacific Rim International Symposium on Dependable Computing, 2002. Proceedings..

[4]  Junfeng Yang,et al.  An empirical study of operating systems errors , 2001, SOSP.

[5]  Isabelle Puaut,et al.  Experimental evaluation of the fail-silent behavior of a distributed real-time run-time support built from COTS components , 2001, 2001 International Conference on Dependable Systems and Networks.

[6]  Ravishankar K. Iyer,et al.  Error sensitivity of the Linux kernel executing on PowerPC G4 and Pentium 4 processors , 2004, International Conference on Dependable Systems and Networks, 2004.

[7]  Ravishankar K. Iyer,et al.  An approach towards benchmarking of fault-tolerant commercial systems , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[8]  Charles P. Shelton,et al.  Robustness testing of the Microsoft Win32 API , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[9]  David A. Patterson,et al.  Towards Availability Benchmarks: A Case Study of Software RAID Systems , 2000, USENIX Annual Technical Conference, General Track.

[10]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[11]  Jean Arlat,et al.  Characterization of the impact of faulty drivers on the robustness of the Linux kernel , 2004, International Conference on Dependable Systems and Networks, 2004.

[12]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[13]  Daniel P. Siewiorek,et al.  Comparing operating systems using robustness benchmarks , 1997, Proceedings of SRDS'97: 16th IEEE Symposium on Reliable Distributed Systems.

[14]  Jeffrey Katcher,et al.  PostMark: A New File System Benchmark , 1997 .

[15]  Jean Arlat,et al.  Benchmarking operating system dependability: Windows 2000 as a case study , 2004, 10th IEEE Pacific Rim International Symposium on Dependable Computing, 2004. Proceedings..

[16]  Jean Arlat,et al.  Analysis of the effects of real and injected software faults: Linux as a case study , 2002, 2002 Pacific Rim International Symposium on Dependable Computing, 2002. Proceedings..

[17]  Philip Koopman,et al.  Comparing the robustness of POSIX operating systems , 1999, Digest of Papers. Twenty-Ninth Annual International Symposium on Fault-Tolerant Computing (Cat. No.99CB36352).

[18]  Jean Arlat,et al.  Benchmarking the dependability of Windows NT4, 2000 and XP , 2004, International Conference on Dependable Systems and Networks, 2004.

[19]  Ji Zhu,et al.  Robustness benchmarking for hardware maintenance events , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[20]  David A. Patterson,et al.  Including the Human Factor in Dependability Benchmarks , 2002 .

[21]  Frank Yellin,et al.  The Java Virtual Machine Specification , 1996 .

[22]  Daniel P. Siewiorek,et al.  Measuring Software Dependability by Robustness Benchmarking , 1997, IEEE Trans. Software Eng..