Checking Formal Specifications by Testing

Formal specification methods hold promise for bridging the wide gap between an intuitive idea for solving a problem by computer, and the executable program that attempts to do the job. The use of formalism is itself a good thing, allowing professionals to understand and analyze their work better. However, formal methods are an aid to human effort, not a panacea. Conventional software testing can be an ideal complement to formally directed development. Tests are concrete and immediately comprehensible to end users, and they are unlikely to miss mistakes because of a pernicious correlation with the formal work. Research is needed on ways to make formal specifications and testing work together to realize the potential of both. Tests should serve to increase confidence that a formal method has been correctly applied. Such tests would free the developers from tedious checking of formalism details, and the success of only a few tests would have real significance for the software’s correctness. As an example of a formalism/testing partnership, this talk describes joint work with Sergio Antoy [4] on automatically checking a conventional implementation of an abstract data type against its formal algebraic specification. 1 Formal Methods and Testing In the “old days” when formal methods were identified with proving an implementation program correct with respect to a first-order logic specification, this proving was thought to be antithetical to testing. The two methods were viewed as being at opposite ends of a spectrum: the formal work was abstract, precise, and concerned with the function the program computes; testing was practical, sloppy, and concerned only with finite support for that function. A more modern viewpoint considers the two subjects complementary. Today’s formal methods are still abstract and precise, but now much more broadly concerned with expressing and reasoning about specifications. Testing today is still practical and sometimes sloppy, but it can be the most technical and precise of the other software-development arts. Specification and testing remain at ∗On leave from Portland State University, Department of Computer Science and Center for Software Quality Research, Portland, OR 97207, USA opposite ends of the development time-line, so there is maximum scope for testing to serve as a check on the whole process. To mention a promising interaction in which testing benefits from a formal specification, consider automatic random testing. This idea is to subject code to inputs chosen at random, where the random generation of test points is planned and automated using a formal description of a program’s input domain. To be effective, random testing must employ far more test points than the usual “by hand” tests. Hence it is practical only if the results can be automatically checked, and here again a formal specification is the answer. Many formal specifications are effective in that they provide a decision procedure for whether or not any input-output pair conforms to the specification. In the other direction, testing has the potential to detect mistakes in software development, mistakes which may arise from poor communication between end users and software experts (that is, the formal specifications are for the wrong problem), and mistakes in carrying through development (that is, the formal specifications have not been properly followed). Here the technical nature of both disciplines comes to the fore: it should be possible to devise tests that pinpoint common mistakes, and if the tests succeed, to gain great confidence that nothing is wrong.