Experience Report: Security Vulnerability Profiles of Mission Critical Software: Empirical Analysis of Security Related Bug Reports

While some prior research work exists on characteristics of software faults (i.e., bugs) and failures, very little work has been published on analysis of software applications vulnerabilities. This paper aims to contribute towards filling that gap by presenting an empirical investigation of application vulnerabilities. The results are based on data extracted from issue tracking systems of two NASA missions. These data were organized in three datasets: Ground mission IV&V issues, Flight mission IV&V issues, and Flight mission Developers issues. In each dataset, we identified the security related software bugs and classified them in specific vulnerability classes. Then, we created the vulnerability profiles, i.e., determined where and when the security vulnerabilities were introduced and what were the dominant vulnerabilities classes. Our main findings include: (1) In IV&V issues datasets the majority of vulnerabilities were code related and were introduced in the Implementation phase. (2) For all datasets, close to 90% of the vulnerabilities were located in two to four subsystems. (3) Out of 21 primary vulnerability classes, five dominated: Exception Management, Memory Access, Other, Risky Values, and Unused Entities. Together, they contributed from around 80% to 90% of vulnerabilities in each dataset.

[1]  Alysson Neves Bessani,et al.  Analysis of operating system diversity for intrusion tolerance , 2014, Softw. Pract. Exp..

[2]  Katerina Goseva-Popstojanova,et al.  Exploring the missing link: an empirical study of software fixes , 2014, Softw. Test. Verification Reliab..

[3]  David Lo,et al.  An Empirical Study of Bugs in Software Build Systems , 2013, 2013 13th International Conference on Quality Software.

[4]  Lorenzo Strigini,et al.  Fault Tolerance via Diversity for Off-the-Shelf Products: A Study with SQL Database Servers , 2007, IEEE Transactions on Dependable and Secure Computing.

[5]  Marco Vieira,et al.  Comparing the Effectiveness of Penetration Testing and Static Code Analysis on the Detection of SQL Injection Vulnerabilities in Web Services , 2009, 2009 15th IEEE Pacific Rim International Symposium on Dependable Computing.

[6]  Indrajit Ray,et al.  Measuring, analyzing and predicting security vulnerabilities in software systems , 2007, Comput. Secur..

[7]  Marco Vieira,et al.  Testing and Comparing Web Vulnerability Scanning Tools for SQL Injection and XSS Attacks , 2007 .

[8]  Norman E. Fenton,et al.  Quantitative Analysis of Faults and Failures in a Complex Software System , 2000, IEEE Trans. Software Eng..

[9]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[10]  Kishor S. Trivedi,et al.  Fault triggers in open-source software: An experience report , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[11]  Nuno Laranjeiro,et al.  Effective Detection of SQL/XPath Injection Vulnerabilities in Web Services , 2009, 2009 IEEE International Conference on Services Computing.

[12]  K. Goseva-Popstojanova,et al.  Common Trends in Software Fault and Failure Data , 2009, IEEE Transactions on Software Engineering.

[13]  Mohammad Zulkernine,et al.  Effective detection of vulnerable and malicious browser extensions , 2014, Comput. Secur..

[14]  Toshinori Sato,et al.  Power-Performance Trade-Off of a Dependable Multicore Processor , 2007 .

[15]  Katerina Goseva-Popstojanova,et al.  Analyzing and predicting effort associated with finding and fixing software faults , 2017, Inf. Softw. Technol..

[16]  Kishor S. Trivedi,et al.  Analysis of bugs in Apache Virtual Computing Lab , 2013, 2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[17]  Marco Vieira,et al.  Analysis of Field Data on Web Security Vulnerabilities , 2014, IEEE Transactions on Dependable and Secure Computing.

[18]  Ravishankar K. Iyer,et al.  Security Vulnerabilities: From Analysis to Detection and Masking Techniques , 2006, Proceedings of the IEEE.

[19]  Marco Vieira,et al.  Mapping software faults with web security vulnerabilities , 2008, 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC (DSN).

[20]  Marco Vieira,et al.  Using web security scanners to detect vulnerabilities in web services , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[21]  Kishor S. Trivedi,et al.  An empirical investigation of fault types in space mission system software , 2010, 2010 IEEE/IFIP International Conference on Dependable Systems & Networks (DSN).

[22]  Katerina Goseva-Popstojanova,et al.  Exploring fault types, detection activities, and failure severity in an evolving safety-critical software system , 2014, Software Quality Journal.

[23]  Kishor S. Trivedi,et al.  An empirical investigation of fault repairs and mitigations in space mission system software , 2013, 2013 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[24]  Laurie A. Williams,et al.  A comparison of the efficiency and effectiveness of vulnerability discovery techniques , 2013, Inf. Softw. Technol..

[25]  Jan H. P. Eloff,et al.  Standardising vulnerability categories , 2008, Comput. Secur..

[26]  Henrique Madeira,et al.  Emulation of Software Faults: A Field Data Study and a Practical Approach , 2006, IEEE Transactions on Software Engineering.

[27]  Ravishankar K. Iyer,et al.  Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[28]  Saurabh Bagchi,et al.  Characterizing Failures in Mobile OSes: A Case Study with Android and Symbian , 2010, 2010 IEEE 21st International Symposium on Software Reliability Engineering.

[29]  Ravishankar K. Iyer,et al.  A data-driven finite state machine model for analyzing security vulnerabilities , 2003, 2003 International Conference on Dependable Systems and Networks, 2003. Proceedings..

[30]  Karthik Pattabiraman,et al.  JavaScript Errors in the Wild: An Empirical Study , 2011, 2011 IEEE 22nd International Symposium on Software Reliability Engineering.

[31]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[32]  Katerina Goseva-Popstojanova,et al.  On the capability of static code analysis to detect security vulnerabilities , 2015, Inf. Softw. Technol..

[33]  Ali Mesbah,et al.  An Empirical Study of Client-Side JavaScript Bugs , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[34]  Gary McGraw,et al.  Seven Pernicious Kingdoms: A Taxonomy of Software Security Errors , 2005, IEEE Secur. Priv..