Cryptographic Verification of Test Coverage Claims

The market for software components is growing, driven on the "demand side" by the need for rapid deployment of highly functional products and, on the "supply side", by distributed object standards. As components and component vendors proliferate, there is naturally a growing concern about quality and the effectiveness of testing processes. White-box testing, particularly the use of coverage criteria, Is a widely used method for measuring the "thoroughness" of testing efforts. High levels of test coverage are used as indicators of good quality control procedures. Software vendors who can demonstrate high levels of test coverage have a credible claim to high quality. However, verifying such claims involves knowledge of the source code, test cases, build procedures, etc. In applications where reliability and quality are critical, it would be desirable to verify test coverage claims without forcing vendors to give up valuable technical secrets. In this paper, we explore cryptographic techniques that can be used to verify such claims. Our techniques have certain limitations, which we discuss in this paper. However, vendors who have done the hard work of developing high levels of test coverage can use these techniques (for a modest additional cost) to provide credible evidence of high coverage, while simultaneously reducing disclosure of intellectual property.