Better testing through oracle selection: (NIER track)

In software testing, the test oracle determines if the application under test has performed an execution correctly. In current testing practice and research, significant effort and thought is placed on selecting test inputs, with the selection of test oracles largely neglected. Here, we argue that improvements to the testing process can be made by considering the problem of oracle selection. In particular, we argue that selecting the test oracle and test inputs together to complement one another may yield improvements testing effectiveness. We illustrate this using an example and present selected results from an ongoing study demonstrating the relationship between test suite selection, oracle selection, and fault finding.

[1]  Steven P. Miller,et al.  Applicability of modified condition/decision coverage to software testing , 1994, Softw. Eng. J..

[2]  Nicolas Halbwachs,et al.  Synchronous Programming of Reactive Systems , 1992, CAV.

[3]  Ajitha Rajan,et al.  The effect of program and model structure on mc/dc test adequacy coverage , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[4]  Debra J. Richardson,et al.  Specification-based test oracles for reactive systems , 1992, International Conference on Software Engineering.

[5]  Jeffrey M. Voas,et al.  Putting assertions in their place , 1994, Proceedings of 1994 IEEE International Symposium on Software Reliability Engineering.

[6]  Massimiliano Di Penta,et al.  Assessing and improving state-based class testing: a series of experiments , 2004, IEEE Transactions on Software Engineering.

[7]  M. Young Test Oracles , 2001 .

[8]  Atif M. Memon,et al.  Designing and comparing automated test oracles for GUI-based software applications , 2007, TSEM.

[9]  Mats Per Erik Heimdahl,et al.  Programs, tests, and oracles: the foundations of testing revisited , 2011, 2011 33rd International Conference on Software Engineering (ICSE).

[10]  Antonia Bertolino,et al.  Software Testing Research: Achievements, Challenges, Dreams , 2007, Future of Software Engineering (FOSE '07).

[11]  Ajitha Rajan,et al.  Coverage metrics for requirements-based testing , 2006, ISSTA '06.

[12]  Mary Jean Harrold,et al.  Testing: a roadmap , 2000, ICSE '00.

[13]  Carl E. Landwehr,et al.  Basic concepts and taxonomy of dependable and secure computing , 2004, IEEE Transactions on Dependable and Secure Computing.