An historical examination of open source releases and their vulnerabilities

This paper examines historical releases of Sendmail, Postfix, Apache httpd and OpenSSL by using static source code analysis and the entry-rate in the Common Vulnerabilities and Exposures dictionary (CVE) for a release, which we take as a measure of the rate of discovery of exploitable bugs. We show that the change in number and density of issues reported by the source code analyzer is indicative of the change in rate of discovery of exploitable bugs for new releases --- formally we demonstrate a statistically significant correlation of moderate strength. The strength of the correlation is an artifact of other factors such as the degree of scrutiny: the number of security analysts investigating the software. This also demonstrates that static source code analysis can be used to make some assessment of risk even when constraints do not permit human review of the issues identified by the analysis. We find only a weak correlation between absolute values measured by the source code analyzer and rate of discovery of exploitable bugs, so in general it is unsafe to use absolute values of number of issues or issue densities to compare different applications or software. Our results demonstrate that software quality, as measured by the number of issues, issue density or number of exploitable bugs, does not always improve with each new release. However, generally the rate of discovery of exploitable bugs begins to drop three to five years after the initial release.