Incremental Identification of Categories and Choices for Test Case Generation: A Study of the Software Practitioners' Preferences

Test case generation is a vital procedure in the engineering of test harnesses. In particular, the choice relation framework and the category-partition method play an important role, by requiring software testers to identify categories (intuitively equivalent to input parameters or environment conditions) and choices (intuitively equivalent to ranges of values) from a specification and to systematically work on the identified choices to generate test cases. Other specification-based test case generation methods (such as the classification-tree method, cause-effect graphing, and combinatorial testing) also have similar requirements, although different terminology such as classifications and classes is used in place of categories and choices. For a large and complex specification that contains many specification components, categories and choices may be identified separately from various kinds of components. We call this practice an incremental identification approach. In this paper, we discuss our study involving 16 experienced software practitioners and three commercial specifications. Our objectives are to determine, from the opinions of the practitioners, (a) the popularity of an incremental identification approach, (b) the usefulness of identifying categories and choices from various kinds of specification components, and (c) possible ways to improve the effectiveness of the identification process.

[1]  Lionel C. Briand.  A Critical Analysis of Empirical Research in Software Testing , 2007, First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007).

[2]  Tsong Yueh Chen,et al.  On the identification of categories and choices for specification-based test case generation , 2004, Inf. Softw. Technol..

[3]  Joc Sanders,et al.  Software quality - a framework for success in software development and support , 1994 .

[4]  Jeffrey C. Carver,et al.  The Impact of Educational Background on the Effectiveness of Requirements Inspections: An Empirical Study , 2008, IEEE Transactions on Software Engineering.

[5]  John Wordsworth.  Software development with Z - a practical approach to formal methods in software engineering , 1992, International computer science series.

[6]  Matthias Grochtmann,et al.  Classification trees for partition testing , 1993, Softw. Test. Verification Reliab..

[7]  Michael L. Fredman,et al.  The AETG System: An Approach to Testing Based on Combinatiorial Design , 1997, IEEE Trans. Software Eng..

[8]  Diane Kelly,et al.  More testing should be taught , 2001, CACM.

[9]  T. H. Tse,et al.  Contributions of tester experience and a checklist guideline to the identification of categories and choices for software testing , 2010, Software Quality Journal.

[10]  Tsong Yueh Chen,et al.  DESSERT: a DividE-and-conquer methodology for identifying categorieS, choiceS, and choicE Relations for Test case generation , 2012, IEEE Transactions on Software Engineering.

[11]  William E. Perry,et al.  Effective methods for software testing , 1995 .

[12]  Richard H. Carver,et al.  A combinatorial testing strategy for concurrent programs , 2007 .

[13]  Tsong Yueh Chen,et al.  A Choice Relation Framework for Supporting Category-Partition Test Case Generation , 2003, IEEE Trans. Software Eng..

[14]  Kishor S. Trivedi,et al.  Fighting bugs: remove, retry, replicate, and rejuvenate , 2007, Computer.

[15]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[16]  Lionel C. Briand,et al.  Using machine learning to refine Category-Partition test specifications and test suites , 2009, Inf. Softw. Technol..

[17]  Adam A. Porter,et al.  Assessing Software Review Meetings: Results of a Comparative Analysis of Two Experimental Studies , 1997, IEEE Trans. Software Eng..

[18]  Barry W. Boehm,et al.  Software Defect Reduction Top 10 List , 2001, Computer.

[19]  Baowen Xu,et al.  A revisit of fault class hierarchies in general boolean specifications , 2011, TSEM.

[20]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[21]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[22]  Man F. Lau,et al.  An extended fault class hierarchy for specification-based testing , 2005, TSEM.

[23]  Walter F. Tichy,et al.  Should Computer Scientists Experiment More? , 1998, Computer.

[24]  Yu Lei,et al.  A Test Generation Strategy for Pairwise Testing , 2002, IEEE Trans. Software Eng..

[25]  Tsong Yueh Chen,et al.  An Integrated Classification-Tree Methodology for Test Case Generation , 2000, Int. J. Softw. Eng. Knowl. Eng..

[26]  Mirko Conrad,et al.  Test case design based on Z and the classification-tree method , 1997, First IEEE International Conference on Formal Engineering Methods.

[27]  Tsong Yueh Chen,et al.  CHOC'LATE: a framework for specification-based testing , 2010, CACM.

[28]  Barry Boehm,et al.  Top 10 list [software development] , 2001 .

[29]  Mark Harman,et al.  Automatically Generating Information from a Z Specification to Support the Classification Tree Method , 2003, ZB.

[30]  Giancarlo Succi,et al.  Effort Prediction in Iterative Software Development Processes -- Incremental Versus Global Prediction Models , 2007, ESEM 2007.