Static Analysis Tool Exposition (SATE) 2008

The NIST SAMATE project conducted the first Static Analysis Tool Exposition (SATE) in 2008 to advance research in static analysis tools that find security defects in source code. The main goals of SATE were to enable empirical research based on large test sets and to encourage improvement and speed adoption of tools. The exposition was planned to be an annual event. Briefly, participating tool makers ran their tool on a set of programs. Researchers led by NIST performed a partial analysis of tool reports. The results and experiences were reported at the Static Analysis Workshop in Tucson, AZ, in June, 2008. The tool reports and analysis were made publicly available in 2009. This paper describes the SATE procedure, provides our observations based on the data collected, and critiques the exposition, including the lessons learned that may help future expositions. This paper also identifies several ways in which the released data and analysis are useful. First, the output from running many tools on production software can be used for empirical research. Second, the analysis of tool reports indicates weaknesses that exist in the software and that are reported by the tools. Finally, the analysis may also be used as a building block for a further study of the weaknesses and of static analysis.