Foundations of Measurement Theory Applied to the Evaluation of Dependability Attributes

Increasing interest is being paid to quantitative evaluation based on measurements of dependability attributes and metrics of computer systems and infrastructures. Despite measurands are generally sensibly identified, different approaches make it difficult to compare different results. Moreover, measurement tools are seldom recognized for what they are: measuring instruments. In this paper, many measurement tools, present in the literature, are critically evaluated at the light of metrology concepts and rules. With no claim of being exhaustive, the paper (i) investigates if and how deeply such tools have been validated in accordance to measurement theory, and (ii) tries to evaluate (if possible) their measurement properties. The intention is to take advantage of knowledge available in a recognized discipline such as metrology and to propose criteria and indicators taken from such discipline to improve the quality of measurements performed in evaluation of dependability attributes.

[1]  F. Dyson.  The inventor of modern science , 1999, Nature.

[2]  Jean Arlat,et al.  Fault Injection for Dependability Validation: A Methodology and Some Applications , 1990, IEEE Trans. Software Eng..

[3]  James Gosling,et al.  The Real-Time Specification for Java , 2000, Computer.

[4]  Péter Urbán,et al.  Neko: a single environment to simulate and prototype distributed algorithms , 2001, Proceedings 15th International Conference on Information Networking.

[5]  William H. Sanders,et al.  Fault injection based on a partial view of the global state of a distributed system , 1999, Proceedings of the 18th IEEE Symposium on Reliable Distributed Systems.

[6]  Johan Karlsson,et al.  GOOFI: generic object-oriented fault injection tool , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[7]  Jean Arlat,et al.  MAFALDA-RT: a tool for dependability assessment of real-time systems , 2002, Proceedings International Conference on Dependable Systems and Networks.

[8]  Farnam Jahanian,et al.  Probing and fault injection of protocol implementations , 1995, Proceedings of 15th International Conference on Distributed Computing Systems.

[9]  Farnam Jahanian,et al.  Testing of fault-tolerant and real-time distributed systems via protocol fault injection , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[10]  Andrea Bondavalli,et al.  Quantitative Evaluation of Distributed Algorithms Using the Neko Framework: The NekoStat Extension , 2005, LADC.

[11]  Flaviu Cristian,et al.  Integrating External and Internal Clock Synchronization , 2004, Real-Time Systems.

[12]  Ravishankar K. Iyer,et al.  An approach towards benchmarking of fault-tolerant commercial systems , 1996, Proceedings of Annual Symposium on Fault Tolerant Computing.

[13]  Amar Phanishayee,et al.  PLATO: Predictive Latency-Aware Total Ordering , 2006, 2006 25th IEEE Symposium on Reliable Distributed Systems (SRDS'06).

[14]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.

[15]  Jean Arlat,et al.  Benchmarking operating system dependability: Windows 2000 as a case study , 2004, 10th IEEE Pacific Rim International Symposium on Dependable Computing, 2004. Proceedings..

[16]  Flaviu Cristian,et al.  FORTRESS: A System to Support Fail-Aware Real-Time Applications , 1997 .

[17]  Ravishankar K. Iyer,et al.  FTAPE: A fault injection tool to measure fault Tolerance , 1994 .

[18]  Henrique Madeira,et al.  Xception: Software Fault Injection and Monitoring in Processor Functional Units1 , 1995 .

[19]  William H. Sanders,et al.  Loki: a state-driven fault injector for distributed systems , 2000, Proceeding International Conference on Dependable Systems and Networks. DSN 2000.

[20]  E. Iso,et al.  Measurement Uncertainty and Probability: Guide to the Expression of Uncertainty in Measurement , 1995 .

[21]  Paulo Veríssimo,et al.  Distributed Systems for System Architects , 2001, Advances in Distributed Computing and Middleware.

[22]  Jean Kovalevsky,et al.  The development of modern metrology and its role today , 2005, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[23]  Kenneth N. Meyer,et al.  Evaluating COTS component dependability in context , 2005, IEEE Software.

[24]  J. Arlat,et al.  Assessment of COTS microkernels by fault injection , 1999, Dependable Computing for Critical Applications 7.