Finding Security Threats That Matter: An Industrial Case Study

Recent trends in the software engineering (i.e., Agile, DevOps) have shortened the development life-cycle limiting resources spent on security analysis of software designs. In this context, architecture models are (often manually) analyzed for potential security threats. Risk-last threat analysis suggests identifying all security threats before prioritizing them. In contrast, risk-first threat analysis suggests identifying the risks before the threats, by-passing threat prioritization. This seems promising for organizations where developing speed is of great importance. Yet, little empirical evidence exists about the effect of sacrificing systematicity for high-priority threats on the performance and execution of threat analysis. To this aim, we conduct a case study with industrial experts from the automotive domain, where we empirically compare a risk-first technique to a risk-last technique. In this study, we consciously trade the amount of participants for a more realistic simulation of threat analysis sessions in practice. This allows us to closely observe industrial experts and gain deep insights into the industrial practice. This work contributes with: (i) a quantitative comparison of performance, (ii) a quantitative and qualitative comparison of execution, and (iii) a comparative discussion of the two techniques. We find no differences in the productivity and timeliness of discovering high-priority security threats. Yet, we find differences in analysis execution. In particular, participants using the risk-first technique found twice as many high-priority threats, developed detailed attack scenarios, and discussed threat feasibility in detail. On the other hand, participants using the risk-last technique found more medium and low-priority threats and finished early.

[1]  Nancy R. Mead,et al.  Software Assurance Competency Model , 2013 .

[2]  M. Losada,et al.  The complex dynamics of high performance teams , 1999 .

[3]  Izak Benbasat,et al.  Institutional pressures in security management: Direct and indirect influences on organizational investment in information security control resources , 2015, Inf. Manag..

[4]  Elissa M. Redmiles,et al.  The Battle for New York: A Case Study of Applied Digital Threat Modeling at the Enterprise Level , 2018, USENIX Security Symposium.

[5]  Wouter Joosen,et al.  A descriptive study of Microsoft’s threat modeling technique , 2015, Requirements Engineering.

[6]  Steve Lipner,et al.  Security development lifecycle , 2010, Datenschutz und Datensicherheit - DuD.

[7]  Andreas L. Opdahl,et al.  Experimental comparison of attack trees and misuse cases for security threat identification , 2009, Inf. Softw. Technol..

[8]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[9]  Fabio Massacci,et al.  An Experimental Comparison of Two Risk-Based Security Methods , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[10]  Andreas L. Opdahl,et al.  Experimental Comparison of Misuse Case Maps with Misuse Cases and System Architecture Diagrams for Eliciting Security Vulnerabilities and Mitigations , 2011, 2011 Sixth International Conference on Availability, Reliability and Security.

[11]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[12]  Riccardo Scandariato,et al.  Threat analysis of software systems: A systematic literature review , 2018, J. Syst. Softw..

[13]  T. Moore,et al.  Identifying How Firms Manage Cybersecurity Investment , 2015 .

[14]  Riccardo Scandariato,et al.  Two Architectural Threat Analysis Techniques Compared , 2018, ECSA.

[15]  Adam Shostack,et al.  Threat Modeling: Designing for Security , 2014 .

[16]  Riccardo Scandariato,et al.  Towards Security Threats that Matter , 2017, CyberICPS/SECPRE@ESORICS.

[17]  Eric Knauss,et al.  Collaborative Traceability Management: Challenges and Opportunities , 2016, 2016 IEEE 24th International Requirements Engineering Conference (RE).

[18]  Guttorm Sindre,et al.  Comparing Misuse Case and Mal-Activity Diagrams for Modelling Social Engineering Attacks , 2012, Int. J. Secur. Softw. Eng..