Towards Empirical Evaluation of Automated Risk Assessment Methods

Security risk assessment methods are numerous, and it might be confusing for organizations to select one. Researchers have conducted empirical studies with established methods in order to find factors that influence their effectiveness and ease of use. In this paper we evaluate the recent TREsPASS semi-automated risk assessment method with respect to the factors identified as critical in several controlled experiments. We also argue that automation of risk assessment raises new research questions that need to be thoroughly investigated in future empirical studies.

[1]  Kim G. Larsen,et al.  Modelling Attack-defense Trees Using Timed Automata , 2016, FORMATS.

[2]  Andreas L. Opdahl,et al.  Comparing attack trees and misuse cases in an industrial setting , 2014, Inf. Softw. Technol..

[3]  Sacha Brostoff,et al.  Transforming the ‘Weakest Link’ — a Human/Computer Interaction Approach to Usable and Effective Security , 2001 .

[4]  Fabio Massacci,et al.  Which security catalogue is better for novices? , 2015, 2015 IEEE Fifth International Workshop on Empirical Requirements Engineering (EmpiRE).

[5]  Eric Li,et al.  From A to Z: Developing a Visual Vocabulary for Information Security Threat Visualisation , 2016, GraMSec@CSF.

[6]  Florian Kammüller,et al.  Transforming Graphical System Models to Graphical Attack Models , 2015, GraMSec@CSF.

[7]  Fabio Massacci,et al.  On the Equivalence Between Graphical and Tabular Representations for Security Risk Assessment , 2017, REFSQ.

[8]  Julie J. C. H. Ryan,et al.  Making Successful Security Decisions: A Qualitative Evaluation , 2012, IEEE Security & Privacy.

[9]  Olga Gadyatskaya,et al.  Attack Trees for Practical Security Assessment: Ranking of Attack Scenarios with ADTool 2.0 , 2016, QEST.

[10]  Olga Gadyatskaya,et al.  Bridging Two Worlds: Reconciling Practical Risk Assessment Methodologies with Theory of Attack Trees , 2016, GraMSec@CSF.

[11]  Ruth Breu,et al.  Evolution of Security Engineering Artifacts: A State of the Art Survey , 2014, Int. J. Secur. Softw. Eng..

[12]  Fabio Massacci,et al.  How to Select a Security Requirements Method? A Comparative Study with Students and Practitioners , 2012, NordSec.

[13]  Wouter Joosen,et al.  A descriptive study of Microsoft’s threat modeling technique , 2015, Requirements Engineering.

[14]  Olga Gadyatskaya,et al.  Using Attack-Defense Trees to Analyze Threats and Countermeasures in an ATM: A Case Study , 2016, PoEM.

[15]  Daniel L. Moody,et al.  The method evaluation model: a theoretical model for validating information systems design methods , 2003, ECIS.

[16]  Wouter Joosen,et al.  Empirical evaluation of a privacy-focused threat modeling methodology , 2014, J. Syst. Softw..

[17]  Fabio Massacci,et al.  An Experimental Comparison of Two Risk-Based Security Methods , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[18]  Jan Willemson,et al.  The Attack Navigator , 2015, GraMSec@CSF.

[19]  Olga Gadyatskaya.  How to Generate Security Cameras: Towards Defence Generation for Socio-Technical Systems , 2015, GraMSec@CSF.

[20]  Fabio Massacci,et al.  An experiment on comparing textual vs. visual industrial methods for security risk assessment , 2014, 2014 IEEE 4th International Workshop on Empirical Requirements Engineering (EmpiRE).

[21]  Tor Stålhane,et al.  Identifying Safety Hazards: An Experimental Comparison of System Diagrams and Textual Use Cases , 2012, BMMDS/EMMSAD.

[22]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[23]  Fabio Massacci,et al.  The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals , 2015, REFSQ.

[24]  Yan Li,et al.  Preliminary Experiments on the Relative Comprehensibility of Tabular and Graphical Risk Models , 2015 .

[25]  Fabio Massacci,et al.  A First Empirical Evaluation Framework for Security Risk Assessment Methods in the ATM Domain , 2014 .

[26]  Ketil Stølen,et al.  Model-Driven Risk Analysis - The CORAS Approach , 2010 .