Practical testing of a C99 compiler using output comparison

A simple technique is presented for testing a C99 compiler, by comparing its output with the output from pre-existing tools. The advantage to this approach is that new test cases can be added in bulk from existing sources, reducing the need for in-depth investigation of correctness issues and for creating new test code by hand. This technique was used in testing the PalmSource Palm OS® Cobalt ARM C/C++ cross-compiler for Palm-Powered® personal digital assistants, primarily for standards compliance and the correct execution of generated code. The technique described here found several hundred bugs, mostly in our in-house code, but also in longstanding high-quality front- and back-end code from Edison Design Group and Apogee Software. It also found 18 bugs in the GNU C compiler, as well as a bug specific to the Apple version of GCC, a bug specific to the Suse version of GCC, and a dozen bugs in versions of GCC for the ARM processor, several of which were critical. Copyright © 2007 John Wiley & Sons, Ltd.