A Threat Analysis Methodology for Security Requirements Elicitation in Machine Learning Based Systems

Machine learning (ML) models are now a key component for many applications. However, machine learning based systems (MLBSs), those systems that incorporate them, have proven vulnerable to various new attacks as a result. Currently, there exists no systematic process for eliciting security requirements for MLBSs that incorporates the identification of adversarial machine learning (AML) threats with those of a traditional non-MLBS. In this research study, we explore the applicability of traditional threat modeling and existing attack libraries in addressing MLBS security in the requirements phase. Using an example MLBS, we examined the applicability of 1) DFD and STRIDE in enumerating AML threats; 2) Microsoft SDL AI/ML Bug Bar in ranking the impact of the identified threats; and 3) the Microsoft AML attack library in eliciting threat mitigations to MLBSs. Such a method has the potential to assist team members, even with only domain specific knowledge, to collaboratively mitigate MLBS threats.

[1]  Xiaohong Yuan,et al.  Developing Abuse Cases Based on Threat Modeling and Attack Patterns , 2015, J. Softw..

[2]  Mesut Ozdag,et al.  Adversarial Attacks and Defenses Against Deep Neural Networks: A Survey , 2018 .

[3]  Jennifer Horkoff,et al.  Non-Functional Requirements for Machine Learning: Challenges and New Directions , 2019, 2019 IEEE 27th International Requirements Engineering Conference (RE).

[4]  M. E. Kabay,et al.  Writing Secure Code , 2015 .

[5]  Elham Tabassi,et al.  A taxonomy and terminology of adversarial machine learning , 2019 .

[6]  Fuyuki Ishikawa,et al.  How Do Engineers Perceive Difficulties in Engineering of Machine-Learning Systems? - Questionnaire Survey , 2019, 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP).

[7]  J. Doug Tygar,et al.  Adversarial machine learning , 2019, AISec '11.

[8]  W. Marsden.  I and J , 2012 .

[9]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[10]  Mohammed Nazir,et al.  A Systematic Review and Analytical Evaluation of Security Requirements Engineering Approaches , 2019 .

[11]  James A. Landay,et al.  Investigating statistical machine learning as a tool for software development , 2008, CHI.

[12]  Andriy V. Miranskyy,et al.  Big Picture of Big Data Software Engineering: With Example Research Challenges , 2015, 2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering.

[13]  Dhirendra Pandey,et al.  STORE: Security Threat Oriented Requirements Engineering Methodology , 2018, Journal of King Saud University - Computer and Information Sciences.

[14]  Frank Swiderski,et al.  Threat Modeling , 2018, Hacking Connected Cars.

[15]  Patrick D. McDaniel,et al.  Making machine learning robust against adversarial inputs , 2018, Commun. ACM.

[16]  Gail C. Murphy,et al.  How does Machine Learning Change Software Development Practices? , 2021, IEEE Transactions on Software Engineering.

[17]  Fan Zhang,et al.  Stealing Machine Learning Models via Prediction APIs , 2016, USENIX Security Symposium.

[18]  Karen A. Scarfone,et al.  The Common Vulnerability Scoring System (CVSS) and its Applicability to Federal Agency Systems , 2007 .

[19]  Michael P. Wellman,et al.  SoK: Security and Privacy in Machine Learning , 2018, 2018 IEEE European Symposium on Security and Privacy (EuroS&P).

[20]  Muhammad Shafique,et al.  Security for Machine Learning-Based Systems: Attacks and Challenges During Training and Inference , 2018, 2018 International Conference on Frontiers of Information Technology (FIT).

[21]  Lei Ma,et al.  Secure Deep Learning Engineering: A Road Towards Quality Assurance of Intelligent Systems , 2019, ICFEM.

[22]  Debdeep Mukhopadhyay,et al.  Adversarial Attacks and Defences: A Survey , 2018, ArXiv.

[23]  Michael N. Johnstone.  Modelling misuse cases as a means of capturing security requirements , 2011 .

[24]  Carlos E. Otero,et al.  Research Directions for Engineering Big Data Analytics Software , 2015, IEEE Intelligent Systems.

[25]  Wei Cai,et al.  A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View , 2018, IEEE Access.

[26]  Adam Shostack,et al.  Threat Modeling: Designing for Security , 2014 .

[27]  Andreas Vogelsang,et al.  Requirements Engineering for Machine Learning: Perspectives from Data Scientists , 2019, 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW).

[28]  R. Stephenson.  A and V , 1962, The British journal of ophthalmology.