An in-depth study of adaptive random testing for testing program with complex input types

Adaptive random testing (ART) is a testing method aimed to improve the performance of random testing. The initial studies of ART merely focussed on programs with numeric input type. However, real life software involves more complex types of inputs. This thesis examines the application of ART on programs with complex input types. Firstly, the application of a distance-based ART technique, namely FSCS-ART, on real programs with complex input types is conducted. This is an extension of two previous studies which developed new approaches to apply FSCS-ART on such programs. However, those studies have not applied the approaches to test any real programs. Here, they are applied to test four real programs with such input types. An exclusion-based ART, namely Restricted Random Testing (RRT), has been demostrated effective to test programs with numeric input type. We are motivated to also apply RRT on programs with complex input types. Some new approaches are introduced to adjust the use of RRT on such programs. Previous studies of ART have identified the regularity of failure causing inputs (FCIs), which are inputs that cause a program under test fails, in the numeric context. They have also observed how ART techniques behaved towards the identified patterns. This thesis investigates the extension of the study to more complex input types. Using the results of the relevant experiments in this thesis, we investigate the FCI patterns in a program with complex input types and analyse how the ART techniques behave towards the identified patterns. When applying ART, the existence of a test oracle is required to identify the detection of failures. Test oracle is more difficult to obtain when the program under test has more complex input types. Therefore, in this thesis we propose the use of Metamorphic Testing (MT) as the test oracle for applying ART on a program with complex input types. We examine how effective is MT compared to a test oracle which has been commonly used in previous ART studies. At last, an application of ART on other area of testing is conducted. We apply ART as the source test case selection strategy of MT. We investigate how effective ART compared to RT in selecting the source test cases of the implemented metamorphic relations.

[1]  Tsong Yueh Chen,et al.  Case studies on the selection of useful relations in metamorphic testing , 2004 .

[2]  Robert G. Merkel,et al.  Analysis and enhancements of adaptive random testing , 2005 .

[3]  W. Chan,et al.  A Metamorphic Testing Approach for Online Testing of Service-Oriented Software Applications , 2007, Int. J. Web Serv. Res..

[4]  Tsong Yueh Chen,et al.  Mirror adaptive random testing , 2004, Inf. Softw. Technol..

[5]  Marc J. Balcer,et al.  The category-partition method for specifying and generating fuctional tests , 1988, CACM.

[6]  Carl K. Chang,et al.  Practical Extensions of a Randomized Testing Tool , 2009, 2009 33rd Annual IEEE International Computer Software and Applications Conference.

[7]  Bertrand Meyer,et al.  ARTOO , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[8]  Yu Liu,et al.  An Experimental Evaluation of the Reliability of Adaptive Random Testing Methods , 2008, 2008 Second International Conference on Secure System Integration and Reliability Improvement.

[9]  Arnaud Gotlieb.  Exploiting symmetries to test programs , 2003, 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003..

[10]  Simeon C. Ntafos,et al.  An Evaluation of Random Testing , 1984, IEEE Transactions on Software Engineering.

[11]  Joe W. Duran,et al.  Quantifying Software Validity by Sampling , 1980, IEEE Transactions on Reliability.

[12]  Donald R. Slutz,et al.  Massive Stochastic Testing of SQL , 1998, VLDB.

[13]  Tsong Yueh Chen,et al.  On the statistical properties of testing effectiveness measures , 2006, J. Syst. Softw..

[14]  Jianjun Zhao,et al.  A Divergence-Oriented Approach to Adaptive Random Testing of Java Programs , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[15]  Tsong Yueh Chen,et al.  Proportional sampling strategy: a compendium and some insights , 2001, J. Syst. Softw..

[16]  Sandra Loosemore,et al.  The GNU C Library Reference Manual , 2001 .

[17]  Lionel C. Briand,et al.  Is mutation an appropriate tool for testing experiments? , 2005, ICSE.

[18]  S. L. Gerhart,et al.  Toward a theory of test data selection , 1975, IEEE Transactions on Software Engineering.

[19]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[20]  T. H. Tse,et al.  Adaptive Random Test Case Prioritization , 2009, 2009 IEEE/ACM International Conference on Automated Software Engineering.

[21]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[22]  Wynne Hsu,et al.  DESIGN OF MUTANT OPERATORS FOR THE C PROGRAMMING LANGUAGE , 2006 .

[23]  Elaine J. Weyuker,et al.  On Testing Non-Testable Programs , 1982, Comput. J..

[24]  William E. Howden,et al.  Reliability of the Path Analysis Testing Strategy , 1976, IEEE Transactions on Software Engineering.

[25]  Johannes Mayer,et al.  Lattice-based adaptive random testing , 2005, ASE.

[26]  Clifford J. Maloney,et al.  Systematic mistake analysis of digital computer programs , 1963, CACM.

[27]  I. K. Mak,et al.  Adaptive Random Testing , 2004, ASIAN.

[28]  William Perry.  Effective methods for software testing, third edition , 1995 .

[29]  R. Kirk.  Experimental Design: Procedures for the Behavioral Sciences , 1970 .

[30]  Takahide Yoshikawa,et al.  Random program generator for Java JIT compiler test system , 2003, Third International Conference on Quality Software, 2003. Proceedings..

[31]  Tsong Yueh Chen,et al.  Adaptive random testing by localization , 2004, 11th Asia-Pacific Software Engineering Conference.

[32]  Gregg Rothermel,et al.  Prioritizing test cases for regression testing , 2000, ISSTA '00.

[33]  Tsong Yueh Chen,et al.  Metamorphic testing of programs on partial differential equations: a case study , 2002, Proceedings 26th Annual International Computer Software and Applications.

[34]  Tsong Yueh Chen,et al.  Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing , 2002, ISSTA '02.

[35]  Gregg Rothermel,et al.  An experimental determination of sufficient mutant operators , 1996, TSEM.

[36]  Tsong Yueh Chen,et al.  Quasi-Random Testing , 2005, IEEE Transactions on Reliability.

[37]  Tsong Yueh Chen,et al.  Adaptive random testing through dynamic partitioning , 2004 .

[38]  Tsong Yueh Chen,et al.  On Favourable Conditions for Adaptive Random Testing , 2007, Int. J. Softw. Eng. Knowl. Eng..

[39]  Tsong Yueh Chen,et al.  Fault-based testing without the need of oracles , 2003, Inf. Softw. Technol..

[40]  James Miller,et al.  A Novel Evolutionary Approach for Adaptive Random Testing , 2009, IEEE Transactions on Reliability.

[41]  Dave Towey,et al.  Forgetting Test Cases , 2006, 30th Annual International Computer Software and Applications Conference (COMPSAC'06).

[42]  Stephen S. Yau,et al.  Integration Testing of Context-sensitive Middleware-based Applications: a Metamorphic Approach , 2006, Int. J. Softw. Eng. Knowl. Eng..

[43]  Paul Ammann,et al.  Data Diversity: An Approach to Software Fault Tolerance , 1988, IEEE Trans. Computers.

[44]  William C. Hetzel,et al.  The complete guide to software testing , 1984 .

[45]  Tsong Yueh Chen,et al.  Testing of Heuristic Methods: A Case Study of Greedy Algorithm , 2008, CEE-SET.

[46]  Marc Roper,et al.  Understanding software testing , 1989 .

[47]  Dave Towey,et al.  Restricted Random Testing: Adaptive Random Testing by Exclusion , 2006, Int. J. Softw. Eng. Knowl. Eng..

[48]  Unicom Seminars Ltd,et al.  Software quality assurance, reliability, and testing , 1987 .