A Taxonomy for Test Oracles

Software test automation is often a difficult and complex process. The most familiar aspects of test automation are organizing and running of test cases and capturing and verifying test results. A set of expected results are needed for each test case in order to check the test results. Generation of these expected results is often done using a mechanism called a test oracle. This paper describes classes of oracles for various types of automated software verification and validation. Several relevant characteristics of oracles are included and the advantages and disadvantages for each class covered. Background Software testing is a process of providing inputs to software under test (SUT) and evaluating the results. In software testing, the mechanism used to generate expected results is called an oracle. (In this paper, the first letter will be capitalized when referring to an Oracle for a specific test.) Many different approaches can be used to generate, capture, and compare test results. The author, for example, at one time or another has used the following methods for generating expected results: