Modeling and Verifying Combinatorial Interactions to Test Data Intensive Systems: Experience at the Norwegian Customs Directorate

Data-intensive systems in e-governance collect and process data to ensure conformance to a set of business rules. Testers meticulously verify data in test databases, extracted from different steps of a live production stream , for correct application of business rules. We simplify the process by allowing testers to model a test domain on a relational database and automatically generate test cases representing data interactions satisfying combinatorial interaction coverage criteria. This paper also introduces test cases with self-referential interactions, which is a necessity in real-world databases. We verify these test cases using our human-in-the-loop tool, Depict. Depict, with expert assistance, generates complex SQL queries for test cases and produces a visual report of test case satisfaction. We apply the approach to two scenarios: 1) simplify and optimize a periodic archiving operation and 2) verify fault codes within the testing environment of the Custom directorate's TVINN system.

[1]  Dan Rubel.  The Heart of Eclipse , 2006, ACM Queue.

[2]  Phyllis Reisner,et al.  Human Factors Studies of Database Query Languages: A Survey and Assessment , 1981, CSUR.

[3]  Jennifer Widom,et al.  Performance Issues in Incremental Warehouse Maintenance , 2000, VLDB.

[4]  Sarfraz Khurshid,et al.  Automated SQL query generation for systematic testing of database engines , 2010, ASE.

[5]  Peter M. Kruse,et al.  Automating test case design within the classification tree editor , 2014, 2014 Federated Conference on Computer Science and Information Systems.

[6]  Javier Tuya,et al.  Using an SQL coverage measurement for testing database applications , 2004, SIGSOFT '04/FSE-12.

[7]  Javier Tuya,et al.  Full predicate coverage for testing SQL database queries , 2010, Softw. Test. Verification Reliab..

[8]  J. Wegener,et al.  Test Case Design by Means of the CTE XL , 2000 .

[9]  Mary Lou Soffa,et al.  Database-aware test coverage monitoring , 2008, ISEC '08.

[10]  Arnab Sarkar,et al.  Experience Report: Verifying Data Interaction Coverage to Improve Testing of Data-Intensive Systems: The Norwegian Customs and Excise Case Study , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[11]  Michal Young,et al.  Residual test coverage monitoring , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[12]  Gregg Rothermel,et al.  WYSIWYT testing in the spreadsheet paradigm: an empirical evaluation , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[13]  Surajit Chaudhuri,et al.  Flexible Database Generators , 2005, VLDB.

[14]  Jeffrey K. Hollingsworth,et al.  Efficient instrumentation for code coverage testing , 2002, ISSTA '02.

[15]  Arnaud Gotlieb,et al.  Modelling data interaction requirements: A position paper , 2013, 2013 3rd International Workshop on Model-Driven Requirements Engineering (MoDRE).

[16]  Hareton K. N. Leung,et al.  The Minimal Failure-Causing Schema of Combinatorial Testing , 2011, TSEM.

[17]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[18]  Jacques Klein,et al.  Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[19]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[20]  David W. Stemple,et al.  Human factors comparison of a procedural and a nonprocedural query language , 1981, TODS.

[21]  Robert Brownlie,et al.  Robust testing of AT&T PMX/StarMAIL using OATS , 1992, AT&T Technical Journal.

[22]  Tao Xie,et al.  Guided test generation for database applications via synthesized database interactions , 2014, ACM Trans. Softw. Eng. Methodol..

[23]  Yashwant K. Malaiya,et al.  Automatic test generation using checkpoint encoding and antirandom testing , 1997, Proceedings The Eighth International Symposium on Software Reliability Engineering.

[24]  D. Richard Kuhn,et al.  Software fault interactions and implications for software testing , 2004, IEEE Transactions on Software Engineering.

[25]  Arnaud Gotlieb,et al.  Testing a Data-Intensive System with Generated Data Interactions - The Norwegian Customs and Excise Case Study , 2013, CAiSE.

[26]  Stéphane Bressan,et al.  Introduction to Database Systems , 2005 .

[27]  Wenfei Fan,et al.  Dependencies revisited for improving data quality , 2008, PODS.

[28]  Charles J. Colbourn,et al.  Covering arrays of higher strength from permutation vectors , 2006 .

[29]  A. Jefferson Offutt,et al.  Combination testing strategies: a survey , 2005, Softw. Test. Verification Reliab..

[30]  Sarfraz Khurshid,et al.  Query-Aware Test Generation Using a Relational Constraint Solver , 2008, 2008 23rd IEEE/ACM International Conference on Automated Software Engineering.

[31]  Mary Lou Soffa,et al.  A family of test adequacy criteria for database-driven applications , 2003, ESEC/FSE-11.

[32]  Wenfei Fan,et al.  Conditional functional dependencies for capturing data inconsistencies , 2008, TODS.

[33]  Rico Wind,et al.  Simple and realistic data generation , 2006, VLDB.

[34]  Robert Mandl,et al.  Orthogonal Latin squares: an application of experiment design to compiler testing , 1985, CACM.

[35]  Raúl A. Santelices,et al.  Efficiently monitoring data-flow test coverage , 2007, ASE.

[36]  Yi Zhang,et al.  MODA: automated test generation for database applications via mock objects , 2010, ASE '10.

[37]  Tao Xie,et al.  Program-input generation for testing database applications using existing database states , 2015, Automated Software Engineering.

[38]  Surajit Chaudhuri,et al.  Generating Queries with Cardinality Constraints for DBMS Testing , 2006, IEEE Transactions on Knowledge and Data Engineering.

[39]  Bruno Legeard,et al.  A taxonomy of model-based testing , 2006 .

[40]  Lionel C. Briand,et al.  Industrial experiences with automated regression testing of a legacy database application , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[41]  Joachim Wegener,et al.  Test Sequence Generation from Classification Trees , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[42]  Yu Lei,et al.  In-parameter-order: a test generation strategy for pairwise testing , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[43]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[44]  Arnab Sarkar,et al.  Evaluating Reconfiguration Impact in Self-Adaptive Systems -- An Approach Based on Combinatorial Interaction Testing , 2015, 2015 41st Euromicro Conference on Software Engineering and Advanced Applications.

[45]  M.A. Vouk,et al.  On effectiveness of pairwise methodology for testing network-centric software , 2005, 2005 International Conference on Information and Communication Technology.

[46]  Myra B. Cohen,et al.  Covering arrays for efficient fault characterization in complex configuration spaces , 2004, IEEE Transactions on Software Engineering.

[47]  Suzanne M. Embury,et al.  An intensional approach to the specification of test cases for database applications , 2006, ICSE '06.

[48]  Mary Lou Soffa,et al.  Regression testing of GUIs , 2003, ESEC/FSE-11.

[49]  Hongjun Lu,et al.  A survey on usage of SQL , 1993, SGMD.

[50]  J. Czerwonka.  Pairwise Testing in the Real World : Practical Extensions to Test-Case Scenarios , 2011 .

[51]  Graham Hamilton,et al.  JDBC API Tutorial and Reference, Second Edition: Universal Data Access for the Java 2 Platform , 1999 .

[52]  Meikel Pöss,et al.  Generating Thousand Benchmark Queries in Seconds , 2004, VLDB.

[53]  Alessandro Orso,et al.  Command-Form Coverage for Testing Database Applications , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).