Evaluating the “ Small Scope Hypothesis ”

The “small scope hypothesis” argues that a high proportion of bugs can be found by testing the program for all test inputs within some small scope. In object-oriented pro grams, a test input is constructed from objects of different classes; a test input is within a scope of s if at mosts objects of any given class appear in it. If the hypothesis holds , it follows that it is more effective to do systematic testing within a small scope than to generate fewer test inputs of a larger scope. This paper evaluates the hypothesis for several implementations of data structures, including some from the Java Collections Framework. We measure how statement coverage, branch coverage, and rate of mutant killing vary with scope. For systematic input generation and correctness checking of Java programs, we use the Korat framework. This paper also presents the Ferastrau framework that we have developed for mutation testing of Java programs. The experimental results show that exhaustive testing within small scopes can achieve complete coverage and kill most of the mutants, even for intricate methods that manipulate complex data structures. The results also show tha t Korat can be used effectively to generate inputs and check correctness for these scopes.