Logging to Facilitate Combinatorial System Testing

Testing a web application is typically very complicated. Imposing simple coverage criteria such as function or line coverage is often not sufficient to uncover bugs due to incorrect components integration. Combinatorial testing can enforce a stronger criterion, while still allowing the prioritization of test cases in order to keep the overall effort feasible. Combinatorial testing requires the whole testing domain to be classified and formalized, e.g., in terms of classification trees. At the system testing level, these trees can be quite large. This short paper presents our preliminary work to automatically construct classification trees from loggings of the system, and to subsequently calculate the coverage of our test runs against various combinatorial criteria. We use the tool CTE which allows such criteria to be custom specified. Furthermore, it comes with a graphical interface to simplify the specification of new test sequences.

[1]  Stephen McCamant,et al.  The Daikon system for dynamic detection of likely invariants , 2007, Sci. Comput. Program..

[2]  Hareton K. N. Leung,et al.  A survey of combinatorial testing , 2011, CSUR.

[3]  Martin S. Feather.  Rapid Application of Lightweight Formal Methods for Consistency Analysis , 1998, IEEE Trans. Software Eng..

[4]  M. S. Feather.  Rapid application of lightweight formal methods for consistency analyses : Managing inconsistency in software development , 1998 .

[5]  R. B. Misra,et al.  Software operational profile based test case allocation using fuzzy logic , 2007, Int. J. Autom. Comput..

[6]  Leonardo Mariani,et al.  Automatic generation of software behavioral models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[7]  Peter M. Kruse,et al.  Automated Test Case Generation Using Classification Trees , 2010 .

[8]  Mark Harman,et al.  Finding the Optimal Balance between Over and Under Approximation of Models Inferred from Execution Logs , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[9]  Mirko Conrad,et al.  Model-based Generation and Structured Representation of Test Scenarios , 1999 .

[10]  Ståle Amland.  Risk-based testing: : Risk analysis fundamentals and metrics for software testing including a financial application case study , 2000, J. Syst. Softw..

[11]  Mark Harman,et al.  Automatically Generating Information from a Z Specification to Support the Classification Tree Method , 2003, ZB.

[12]  Paolo Tonella,et al.  State-Based Testing of Ajax Web Applications , 2008, 2008 1st International Conference on Software Testing, Verification, and Validation.

[13]  Mary Lou Soffa,et al.  Coverage criteria for GUI testing , 2001, ESEC/FSE-9.

[14]  Jesse H. Poore,et al.  Statistical testing of software based on a usage model , 1995, Softw. Pract. Exp..

[15]  Zhi-Li Zhang,et al.  Extracting the textual and temporal structure of supercomputing logs , 2009, 2009 International Conference on High Performance Computing (HiPC).

[16]  Myra B. Cohen,et al.  Interaction testing of highly-configurable systems in the presence of constraints , 2007, ISSTA '07.

[17]  Mirko Conrad,et al.  Test case design based on Z and the classification-tree method , 1997, First IEEE International Conference on Formal Engineering Methods.

[18]  John D. Musa,et al.  Operational profiles in software-reliability engineering , 1993, IEEE Software.

[19]  Jonathan P. Bowen,et al.  ZB 2003: Formal Specification and Development in Z and B , 2003, Lecture Notes in Computer Science.

[20]  AmlandStåle.  Risk-based testing , 2000 .

[21]  Stéphane Ducasse,et al.  Object-oriented legacy system trace-based logic testing , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[22]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[23]  Paolo Tonella,et al.  Combining model-based and combinatorial testing for effective test case generation , 2012, ISSTA 2012.

[24]  Gregg Rothermel,et al.  Test Case Prioritization: A Family of Empirical Studies , 2002, IEEE Trans. Software Eng..

[25]  Joachim Wegener,et al.  Test Sequence Generation from Classification Trees , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.