Finding Faults Quickly in Formal Models using Random Search

As software grows more complex, automatic verification tools become increasingly important. Unfortunately many systems are large enough that complete verification requires a lot of time and memory, if it is possible at all. In our preliminary studies, random search, although not a complete technique, was able to find most faults significantly faster and with less memory than would be required for full verification. Here we present an experiment in which random search was used to find faults in fault-seeded models of a large commercial flight guidance system. To assess the performance of random search we compared it to a full verification done by the model checker NuSMV. The random search results were surprisingly complete, finding nearly 90% of the faults reported by NuSMV—and these results were generated faster and using less memory. We suggest that random search be used in conjunction with verification tools, perhaps as a fast debugging tool during model development, or even as an alternative model checking strategy on models for which the time and memory requirements would make full verification impossible.