Resilience Benchmarking of Transactional Systems: Experimental Study of Alternative Metrics

Assessing and comparing computer systems under changing contexts is becoming crucial due to the dynamic characteristics of modern computing environments. This is especially relevant for database management systems, as the behavior of the DBMS when immersed in today's volatile environments is determinant for the success of a multitude of commercial, industrial and scientific endeavors. This paper presents an example of a complete resilience benchmarking scenario for database centric systems and concrete benchmark results, showing that the concept of resilience benchmarking is sound and highly applicable to real transactional systems. The key elements of the approach are discussed, and we define a procedure and a changeload, which includes a set of typical changes that affect the available resources (e.g. memory, CPU) and variations in the type of transactions executed. We then use three distinct quantification approaches to evaluate the resilience of the considered transactional systems concerning their performance, and draw conclusions on the suitableness of the considered metrics for resilience benchmarking.

[1]  Marc Holze,et al.  Towards workload-aware self-management: Predicting significant workload shifts , 2010, 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010).

[2]  Yanpei Chen,et al.  From TPC-C to Big Data Benchmarks: A Functional Workload Model , 2012, WBDB.

[3]  Marco Vieira,et al.  Changeloads: A Fundamental Piece on the SASO Systems Benchmarking Puzzle , 2012, 2012 IEEE Sixth International Conference on Self-Adaptive and Self-Organizing Systems Workshops.

[4]  Luca Simoncini.  Resilient computing: An engineering discipline , 2009, 2009 IEEE International Symposium on Parallel & Distributed Processing.

[5]  David Woods,et al.  Resilience Engineering: Concepts and Precepts , 2006 .

[6]  A. Avritzer,et al.  Fault tolerance and resilience : meanings , measures and assessment , 2012 .

[7]  Said Elnaffar,et al.  Automatically classifying database workloads , 2002, CIKM '02.

[8]  Robert Dorbritz.  ASSESSING THE RESILIENCE OF TRANSPORTATION SYSTEMS IN CASE OF LARGE-SCALE DISASTROUS EVENTS , 2011 .

[9]  Jean-Claude Laprie,et al.  From Dependability to Resilience , 2008, DSN 2008.

[10]  Kun Ji,et al.  Resilient industrial control system (RICS): Concepts, formulation, metrics, and insights , 2010, 2010 3rd International Symposium on Resilient Control Systems.

[11]  Jack Demick,et al.  Theory and Measurement of Resilience: Views from Development , 2012 .

[12]  Michel Bruneau,et al.  Conceptualizing and measuring resilience: A key to disaster loss reduction , 2007 .

[13]  Yves Crouzet,et al.  System Dependability: Characterization and Benchmarking , 2012, Adv. Comput..

[14]  Dong Seong Kim,et al.  Resilience in computer systems and networks , 2009, 2009 IEEE/ACM International Conference on Computer-Aided Design - Digest of Technical Papers.

[15]  Marco Vieira,et al.  SCoRe: An across-the-board metric for computer systems resilience benchmarking , 2013, 2013 43rd Annual IEEE/IFIP Conference on Dependable Systems and Networks Workshop (DSN-W).

[16]  Carlo Curino,et al.  OLTP-Bench: An Extensible Testbed for Benchmarking Relational Databases , 2013, Proc. VLDB Endow..

[17]  Min Ouyang,et al.  A three-stage resilience analysis framework for urban infrastructure systems , 2012 .

[18]  Marco Vieira,et al.  Changeloads for Resilience Benchmarking of Self-Adaptive Systems: A Risk-Based Approach , 2012, 2012 Ninth European Dependable Computing Conference.