Experimental comparison of attack trees and misuse cases for security threat identification

A number of methods have been proposed or adapted to include security in the requirements analysis stage, but the industrial take-up has been limited and there are few empirical and comparative evaluations. This paper reports on a pair of controlled experiments that compared two methods for early elicitation of security threats, namely attack trees and misuse cases. The 28 and 35 participants in the two experiments solved two threat identification tasks individually by means of the two techniques, using a Latin-Squares design to control for technique and task order. The dependent variables were effectiveness of the techniques measured as the number of threats found, coverage of the techniques measured in terms of the types of threats found and perceptions of the techniques measured through a post-task questionnaire based on the Technology Acceptance Model. The only difference was that, in the second experiment, the participants were given a pre-drawn use-case diagram to use as a starting point for solving the tasks. In the first experiment, no pre-drawn use-case diagram was provided. The main finding was that attack trees were more effective for finding threats, in particular when there was no pre-drawn use-case diagram. However, the participants had similar opinions of the two techniques, and perception of a technique was not correlated with performance with that technique. The study underlines the need for further comparisons in a broader range of settings involving additional techniques, and it suggests several concrete experiments and other paths for further work.