Testing DBMS Performance with Mutations

Database systems power modern data-intensive applications. It is important to ensure that they operate correctly. To this end, developers extensively test these systems to eliminate bugs that negatively affect functionality. Besides functional bugs, there is another important class of bugs that negatively affect the response time of a database system, known as performance bugs. Despite their impact on enduser experience, performance bugs have received considerably less attention than functional bugs. In this paper, we present AMOEBA, a system for automatically detecting performance bugs in database systems. The core idea behind AMOEBA is to construct query pairs that are semantically equivalent to each other and then compare their response time on the same database system. If the queries exhibit a significant difference in their runtime performance, then the root cause is likely a performance bug in the system. We propose a novel set of mutation rules for constructing query pairs that are likely to uncover performance bugs. We introduce feedback mechanisms for improving the efficacy and computational efficiency of the tool. We evaluate AMOEBA on two widely-used DBMSs, namely PostgreSQL and CockroachDB. AMOEBA has so far discovered 39 previously-unknown performance bugs, among which developers have already confirmed 14 bugs and fixed 4 bugs.

[1]  S. Sudarshan,et al.  Generating test data for killing SQL mutants: A constraint-based approach , 2011, 2011 IEEE 27th International Conference on Data Engineering.

[2]  Gerhard Weikum,et al.  Self-tuning Database Technology and Information Services: from Wishful Thinking to Viable Engineering , 2002, VLDB.

[3]  Goetz Graefe,et al.  Query evaluation techniques for large databases , 1993, CSUR.

[4]  Dinghao Wu,et al.  SQUIRREL: Testing Database Management Systems with Language Validity and Coverage Feedback , 2020, CCS.

[5]  Hicham G. Elmongui,et al.  A framework for testing query transformation rules , 2009, SIGMOD Conference.

[6]  Leo Giakoumakis,et al.  A genetic approach for random testing of database systems , 2007, VLDB.

[7]  Zhendong Su,et al.  Detecting optimization bugs in database engines via non-optimizing reference engine construction , 2020, ESEC/SIGSOFT FSE.

[8]  César A. Galindo-Legaria,et al.  Testing SQL Server's Query Optimizer: Challenges, Techniques and Experiences , 2008, IEEE Data Eng. Bull..

[9]  Raghu Ramakrishnan,et al.  Database Management Systems , 1976 .

[10]  Zhendong Su,et al.  Finding bugs in database systems via query partitioning , 2020, Proc. ACM Program. Lang..

[11]  Mohamed A. Soliman,et al.  Testing the accuracy of query optimizers , 2012, DBTest '12.

[12]  Alan Fekete,et al.  A teaching system for SQL , 1997, ACSE '97.

[13]  William R. Harris,et al.  Automated Verification of Query Equivalence Using Satisfiability Modulo Theories , 2019, Proc. VLDB Endow..

[14]  Tsong Yueh Chen,et al.  Metamorphic Testing: A New Approach for Generating Next Test Cases , 2020, ArXiv.

[15]  Surajit Chaudhuri,et al.  Generating Queries with Cardinality Constraints for DBMS Testing , 2006, IEEE Transactions on Knowledge and Data Engineering.

[16]  W. M. McKeeman,et al.  Differential Testing for Software , 1998, Digit. Tech. J..

[17]  Zhan Li,et al.  OptMark: A Toolkit for Benchmarking Query Optimizers , 2016, CIKM.

[18]  Viktor Leis,et al.  How Good Are Query Optimizers, Really? , 2015, Proc. VLDB Endow..

[19]  Zhendong Su,et al.  Testing Database Engines via Pivoted Query Synthesis , 2020, OSDI.

[20]  Jaroslav Pokorný,et al.  Database technologies in the world of big data , 2015, CompSysTech '15.

[21]  Dong Hun Lee,et al.  Performance Monitoring in SAP HANA's Continuous Integration Process , 2016, PERV.

[22]  William J. Knottenbelt,et al.  Database system performance evaluation models: A survey , 2012, Perform. Evaluation.

[23]  Toshihide Ibaraki,et al.  On the optimal nesting order for computing N-relational joins , 1984, TODS.

[24]  Alvin Cheung,et al.  Cosette: An Automated Prover for SQL , 2017, CIDR.

[25]  Hamid Pirahesh,et al.  Extensible/rule based query rewrite optimization in Starburst , 1992, SIGMOD '92.

[26]  Taesoo Kim,et al.  APOLLO: Automatic Detection and Diagnosis of Performance Regressions in Database Systems , 2019, Proc. VLDB Endow..

[27]  Benoît Dageville,et al.  Oracle's SQL Performance Analyzer , 2008, IEEE Data Eng. Bull..

[28]  C. L. Philip Chen,et al.  Data-intensive applications, challenges, techniques and technologies: A survey on Big Data , 2014, Inf. Sci..

[29]  Donald R. Slutz,et al.  Massive Stochastic Testing of SQL , 1998, VLDB.

[30]  Donghun Lee,et al.  A Performance Anomaly Detection and Analysis Framework for DBMS Development , 2012, IEEE Transactions on Knowledge and Data Engineering.

[31]  Bikash Chandra,et al.  Data generation for testing and grading SQL queries , 2015, The VLDB Journal.

[32]  César A. Galindo-Legaria,et al.  Counting, enumerating, and sampling of execution plans in a cost-based query optimizer , 2000, SIGMOD '00.

[33]  Shrainik Jain,et al.  Snowtrail: Testing with Production Queries on a Cloud Database , 2018, DBTest@SIGMOD.

[34]  Hitesh Kumar Sharma,et al.  Explain Plan and SQL Trace the Two Approaches for RDBMS Tuning , 2017 .