Query-Aware Test Generation Using a Relational Constraint Solver

We present a novel approach for black-box testing of database management systems (DBMS) using the Alloy tool-set. Given a database schema and an SQL query as inputs, our approach first formulates Alloy models for both inputs, and then using the Alloy Analyzer, it generates (1) input data to populate test databases, and (2) the expected result of executing the given query on the generated data. The Alloy Analyzer results form a complete test suite (input/oracle) for verifying the execution result of a DBMS query processor. By incorporating both the schema and the query during the analysis, our approach performs query-aware data generation where executing the query on the generated data produces meaningful non-empty results. We developed a prototype tool, ADUSA, and used it to evaluate our approach. Experimental results show the ability of our approach to detect bugs in both open-source as well as commercial database management systems.

[1]  Daniel Jackson,et al.  Finding bugs with a constraint solver , 2000, ISSTA '00.

[2]  Sarfraz Khurshid,et al.  TestEra: Specification-Based Testing of Java Programs Using SAT , 2004, Automated Software Engineering.

[3]  Rupak Majumdar,et al.  Dynamic test input generation for database applications , 2007, ISSTA '07.

[4]  Sarfraz Khurshid,et al.  Checking Java implementation of a naming architecture using Testera , 2001, Workshop on Software Model Checking @ CAV.

[5]  Emina Torlak,et al.  Kodkod: A Relational Model Finder , 2007, TACAS.

[6]  Yuetang Deng,et al.  Testing database transactions with AGENDA , 2005, Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005..

[7]  Surajit Chaudhuri,et al.  Flexible Database Generators , 2005, VLDB.

[8]  Rico Wind,et al.  Simple and realistic data generation , 2006, VLDB.

[9]  Daniel Jackson,et al.  Software Abstractions - Logic, Language, and Analysis , 2006 .

[10]  Donald R. Slutz,et al.  Massive Stochastic Testing of SQL , 1998, VLDB.

[11]  Carsten Binnig,et al.  QAGen: generating query-aware test databases , 2007, SIGMOD '07.

[12]  Daniel Jackson,et al.  Alloy: a lightweight object modelling notation , 2002, TSEM.

[13]  Yannis Smaragdakis,et al.  Scalable automatic test data generation from modeling diagrams , 2007, ASE.

[14]  Suzanne M. Embury,et al.  An intensional approach to the specification of test cases for database applications , 2006, ICSE '06.

[15]  Meikel Pöss,et al.  Generating Thousand Benchmark Queries in Seconds , 2004, VLDB.

[16]  Surajit Chaudhuri,et al.  Generating Queries with Cardinality Constraints for DBMS Testing , 2006, IEEE Transactions on Knowledge and Data Engineering.

[17]  Daniel Jackson,et al.  Automating first-order relational logic , 2000, SIGSOFT '00/FSE-8.

[18]  Carsten Binnig,et al.  Reverse Query Processing , 2007, 2007 IEEE 23rd International Conference on Data Engineering.