Do bugs foreshadow vulnerabilities? An in-depth study of the chromium project

As developers face an ever-increasing pressure to engineer secure software, researchers are building an understanding of security-sensitive bugs (i.e. vulnerabilities). Research into mining software repositories has greatly increased our understanding of software quality via empirical study of bugs. Conceptually, however, vulnerabilities differ from bugs: they represent an abuse of functionality as opposed to insufficient functionality commonly associated with traditional, non-security bugs. We performed an in-depth analysis of the Chromium project to empirically examine the relationship between bugs and vulnerabilities. We mined 374,686 bugs and 703 post-release vulnerabilities over five Chromium releases that span six years of development. We used logistic regression analysis, ranking analysis, bug type classifications, developer experience, and vulnerability severity metrics to examine the overarching question: are bugs and vulnerabilities in the same files? While we found statistically significant correlations between pre-release bugs and post-release vulnerabilities, we found the association to be weak. Number of features, source lines of code, and pre-release security bugs are, in general, more closely associated with post-release vulnerabilities than any of our non-security bug categories. In further analysis, we examined sub-types of bugs, such as stability-related bugs, and the associations did not improve. Even the files with the most severe vulnerabilities (by measure of CVSS or bounty payouts) did not show strong correlations with number of bugs. These results indicate that bugs and vulnerabilities are empirically dissimilar groups, motivating the need for security engineering research to target vulnerabilities specifically.

[1]  A. Raftery.  Bayesian Model Selection in Social Research , 1995 .

[2]  Mehran Bozorgi,et al.  Beyond heuristics: learning to classify vulnerabilities and predict exploits , 2010, KDD.

[3]  Georgios Gousios,et al.  Dismal Code: Studying the Evolution of Security Bugs , 2013, LASER.

[4]  Ahmed E. Hassan,et al.  Explaining software defects using topic models , 2012, 2012 9th IEEE Working Conference on Mining Software Repositories (MSR).

[5]  Andreas Zeller,et al.  Predicting vulnerable software components , 2007, CCS '07.

[6]  Measuring the Occurrence of Security-Related Bugs through Software Evolution , 2012, 2012 16th Panhellenic Conference on Informatics.

[7]  Michael Gegick,et al.  Prioritizing software security fortification throughcode-level metrics , 2008, QoP '08.

[8]  R. Płoski,et al.  Susceptibility genes in Graves’ ophthalmopathy: searching for a needle in a haystack? , 2007, Clinical endocrinology.

[9]  K. Zakzanis,et al.  Statistics to tell the truth, the whole truth, and nothing but the truth: formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers. , 2001, Archives of clinical neuropsychology : the official journal of the National Academy of Neuropsychologists.

[10]  Eugene H. Spafford,et al.  Software vulnerability analysis , 1998 .

[11]  Audris Mockus,et al.  High-impact defects: a study of breakage and surprise defects , 2011, ESEC/FSE '11.

[12]  J. Ruscio,et al.  A probability-based measure of effect size: robustness to base rates and other factors. , 2008, Psychological methods.

[13]  Koichiro Ochimizu,et al.  Towards logistic regression models for predicting fault-prone code across software projects , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[14]  Koichiro Ochimizu,et al.  Towards logistic regression models for predicting fault-prone code across software projects , 2009, ESEM 2009.

[15]  Indrajit Ray,et al.  Assessing vulnerability exploitability risk using software properties , 2016, Software Quality Journal.

[16]  David R. Anderson,et al.  Multimodel Inference , 2004 .

[17]  Jose J. Gonzalez,et al.  Understanding Hidden Information Security Threats: The Vulnerability Black Market , 2007, 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07).

[18]  Andrew Meneely,et al.  When a Patch Goes Bad: Exploring the Properties of Vulnerability-Contributing Commits , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[19]  Charles Miller,et al.  The Legitimate vulnerability market: the secretive world of 0-day exploit sales , 2007, WEIS.

[20]  Yashwant K. Malaiya,et al.  Comparing and Evaluating CVSS Base Metrics and Microsoft Rating System , 2015, 2015 IEEE International Conference on Software Quality, Reliability and Security.

[21]  Fabio Massacci,et al.  Comparing Vulnerability Severity and Exploits Using Case-Control Studies , 2014, TSEC.

[22]  Arvind K. Tripathi,et al.  Bounty programs in free/libre/open source software , 2006 .

[23]  Michael Gegick,et al.  Predicting Attack-prone Components , 2009, 2009 International Conference on Software Testing Verification and Validation.

[24]  David A. Wagner,et al.  An Empirical Study of Vulnerability Rewards Programs , 2013, USENIX Security Symposium.

[25]  Jacob Cohen.  Statistical Power Analysis , 1992 .

[26]  Antoine Guisan,et al.  Predictive habitat distribution models in ecology , 2000 .

[27]  Tim Menzies,et al.  Past, Present, and Future of Analyzing Software Data , 2015, The Art and Science of Analyzing Software Data.

[28]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[29]  Laurie A. Williams,et al.  Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities , 2011, IEEE Transactions on Software Engineering.

[30]  Norman F. Schneidewind,et al.  Methodology For Validating Software Metrics , 1992, IEEE Trans. Software Eng..

[31]  Fabio Massacci,et al.  Comparing Vulnerability Severity and Exploits Using Case-Control Studies , 2013, TSEC.

[32]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[33]  Alexander Serebrenik,et al.  Process Mining Software Repositories , 2011, 2011 15th European Conference on Software Maintenance and Reengineering.

[34]  Laurie A. Williams,et al.  Searching for a Needle in a Haystack: Predicting Security Vulnerabilities for Windows Vista , 2010, 2010 Third International Conference on Software Testing, Verification and Validation.

[35]  Yashwant K. Malaiya,et al.  Software Vulnerability Markets: Discoverers and Buyers , 2014 .

[36]  Fabio Massacci,et al.  A preliminary analysis of vulnerability scores for attacks in wild: the ekits and sym datasets , 2012, BADGERS@CCS.

[37]  David P. Tegarden,et al.  Effectiveness of traditional software metrics for object-oriented systems , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[38]  Fabio Massacci,et al.  Quantitative Assessment of Risk Reduction with Cybercrime Black Market Monitoring , 2013, 2013 IEEE Security and Privacy Workshops.

[39]  Ken-ichi Matsumoto,et al.  The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[40]  Andrew Meneely,et al.  An empirical investigation of socio-technical code review metrics and security vulnerabilities , 2014, SSE@SIGSOFT FSE.

[41]  Christian Bird,et al.  The Art and Science of Analyzing Software Data , 2015, ICSE 2015.

[42]  M. Mukaka,et al.  Statistics corner: A guide to appropriate use of correlation coefficient in medical research. , 2012, Malawi medical journal : the journal of Medical Association of Malawi.

[43]  Tsong Yueh Chen,et al.  On the statistical properties of the F-measure , 2004, Fourth International Conference onQuality Software, 2004. QSIC 2004. Proceedings..