Formal methods and testing: why the state-of-the art is not the state-of-the practice

Testing is essential for proving the quality of all products. It is expensive and hard to test interactive multimedia systems. This paper proposes a mode] for the test process, and investigates the possibility of deriving test cases from system models and requirement analysis techniques such as requirements definition and specification. We believe this will allow early detection of errors thereby reducing the cost for testing. These test cases can also be used as guidelines on design for testability. Effectiveness of the test tools depends on the careful selection of test cases. I n t r o d u c t i o n The problem of testing software is becoming increasingly difficult as complexity of systems and data volumes grow. It is now frequently the case that 30-50% of development time may be devoted to testing. In particular applications with a graphical or multimedia user interface are difficult to test in an accurate and reproducible manner because of the multiplicity of possible inputs at any time and the volume of data that is output. A traditional software development process consists of a number of stages such as requirements, design, testing and maintenance. During each stage, a number of products have been delivered, but are rarely checked/traced against each other. For example, the object of the testing is to find out the system should never fail and should conform to its requirements. There is no formal way of validating this process because there is no traceability (test matrix) specified in the first place between testing and requirements. In testing there are a number of interesting techniques has emerged which are discussed in Myers (1979). However their effectiveness depend on how we design test cases. It is always difficult for a test group to identify test cases if the group is independent from the development group. One way of solving this cultural situation is to identify clear interactions during the development process right from the beginning. Therefore the test group can work together with development group in order to identify potential test cases. This approach/policy has been successful in several leading organizations. Traditionally, testing has been defined as a process of executing a program with the intent of finding errors, Myers (1979). This is clearly inadequate definition for a complex systems. Therefore we prefer to define testing as follows: