Benchmarking Query Execution Robustness

Benchmarks that focus on running queries on a well-tuned database system ignore a long-standing problem: adverse runtime conditions can cause database system performance to vary widely and unexpectedly. When the query execution engine does not exhibit resilience to these adverse conditions, addressing the resultant performance problems can contribute significantly to the total cost of ownership for a database system in over-provisioning, lost efficiency, and increased human administrative costs. For example, focused human effort may be needed to manually invoke workload management actions or fine-tune the optimization of specific queries. We believe a benchmark is needed to measure query execution robustness, that is, how adverse or unexpected conditions impact the performance of a database system. We offer a preliminary analysis of barriers to query execution robustness and propose some metrics for quantifying the impact of those barriers. We present and analyze results from preliminary tests on four real database systems and discuss how these results could be used to increase the robustness of query processing in each case. Finally, we outline how our efforts could be expanded into a benchmark to quantify query execution robustness.

[1]  David J. DeWitt,et al.  A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment , 1989, SIGMOD '89.

[2]  Volker Markl,et al.  LEO - DB2's LEarning Optimizer , 2001, VLDB.

[3]  Jayant R. Haritsa,et al.  Analyzing Plan Diagrams of Database Query Optimizers , 2005, VLDB.

[4]  Peter J. Haas,et al.  Statistical Learning Techniques for Costing XML Queries , 2005, VLDB.

[5]  Raghunath Othayoth Nambiar,et al.  The making of TPC-DS , 2006, VLDB.

[6]  Goetz Graefe,et al.  Optimization of dynamic query evaluation plans , 1994, SIGMOD '94.

[7]  Hamid Pirahesh,et al.  Robust query processing through progressive optimization , 2004, SIGMOD '04.

[8]  Jayant R. Haritsa,et al.  Identifying robust plans through plan diagram reduction , 2008, Proc. VLDB Endow..

[9]  Harumi A. Kuno,et al.  Dynamic Workload Management for Very Large Data Warehouses: Juggling Feathers and Bowling Balls , 2007, VLDB.

[10]  Ashraf Aboulnaga,et al.  Robustness in automatic physical database design , 2008, EDBT '08.

[11]  Harumi A. Kuno,et al.  Visualizing the robustness of query execution , 2009, CIDR.

[12]  Jennifer Widom,et al.  Content-Based Routing: Different Plans for Different Data , 2005, VLDB.

[13]  Meikel Pöss,et al.  New TPC benchmarks for decision support and web commerce , 2000, SGMD.

[14]  Timos K. Sellis,et al.  Parametric query optimization , 1992, The VLDB Journal.

[15]  Volker Markl,et al.  Learning table access cardinalities with LEO , 2002, SIGMOD '02.

[16]  Jayant R. Haritsa,et al.  On the Production of Anorexic Plan Diagrams , 2007, VLDB.

[17]  Goetz Graefe,et al.  Query evaluation techniques for large databases , 1993, CSUR.

[18]  Jayant R. Haritsa,et al.  Green Query Optimization: Taming Query Optimization Overheads through Plan Recycling , 2004, VLDB.

[19]  Karen C. Davis,et al.  Performance comparison of property map and bitmap indexing , 2002, DOLAP '02.

[20]  David J. DeWitt,et al.  Proactive re-optimization with Rio , 2005, SIGMOD '05.