An Empirical Evaluation of the Effectiveness of Attack Graphs and Fault Trees in Cyber-Attack Perception

Perceiving and understanding cyber-attacks can be a difficult task. This problem is widely recognized and well-documented, and more effective techniques are needed to aid cyber-attack perception. Attack modeling techniques (AMTs), such as attack graphs and fault trees, are useful visual aids that can aid cyber-attack perception; however, there is little empirical or comparative research which evaluates the effectiveness of these methods. This paper reports the results of an empirical evaluation between an adapted attack graph method and the fault tree standard to determine which of the two methods is more effective in aiding cyber-attack perception. An empirical evaluation (<inline-formula> <tex-math notation="LaTeX">$n=63$ </tex-math></inline-formula>) was conducted through a <inline-formula> <tex-math notation="LaTeX">$3 \times 2 \times 2$ </tex-math></inline-formula> factorial design. Participants from computer-science and non-computer-science backgrounds were divided into an adapted attack graph and fault tree group and then asked to complete three tests which tested the ability to recall, comprehend, and apply the AMT. A mean assessment score (<italic>mas</italic>) was calculated for each test. The results show that the adapted attack graph method is more effective at aiding cyber-attack perception when compared with the fault tree method (<inline-formula> <tex-math notation="LaTeX">$p < 0.01$ </tex-math></inline-formula>). Participants that have a computer science background outperformed other participants when using both methods (<inline-formula> <tex-math notation="LaTeX">$p < 0.05$ </tex-math></inline-formula>). These results indicate that the adapted attack graph method can be an effective tool for aiding cyber-attack perception amongst experts. This paper underlines the need for further comparisons in a broader range of settings involving additional techniques, and makes several suggestions for further work.

[1]  Xxyyzz,et al.  Bloom’s Taxonomy , 2020 .

[2]  Stefano Bistarelli,et al.  Strategic Games on Defense Trees , 2006, Formal Aspects in Security and Trust.

[3]  William L. Fithen,et al.  Formal modeling of vulnerability , 2004, Bell Labs Technical Journal.

[4]  Dianxiang Xu,et al.  Security test generation using threat trees , 2009, 2009 ICSE Workshop on Automation of Software Test.

[5]  T. Tidwell,et al.  Modeling Internet Attacks , 2022 .

[6]  S. Nanda,et al.  A highly scalable model for network attack identification and path prediction , 2007, Proceedings 2007 IEEE SoutheastCon.

[7]  Mass Soldal Lund,et al.  How Good are Attack Trees for Modelling Advanced Cyber Threats , 2014 .

[8]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[9]  Soumya K. Ghosh,et al.  Efficient generation of exploit dependency graph by customized attack modeling technique , 2012, 2012 18th International Conference on Advanced Computing and Communications (ADCOM).

[10]  Cynthia A. Phillips,et al.  A graph-based system for network-vulnerability analysis , 1998, NSPW '98.

[11]  Steven J. Templeton,et al.  A requires/provides model for computer attacks , 2001, NSPW '00.

[12]  R. Cunningham,et al.  Validating and Restoring Defense in Depth Using Attack Graphs , 2006, MILCOM 2006 - 2006 IEEE Military Communications conference.

[13]  Eric Michael Hutchins,et al.  Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains , 2010 .

[14]  Sushil Jajodia,et al.  Topological analysis of network attack vulnerability , 2006, PST.

[15]  Christoph Meinel,et al.  A New Alert Correlation Algorithm Based on Attack Graph , 2011, CISIS.

[16]  Igor Nai Fovino,et al.  Integrating cyber attacks within fault trees , 2009, Reliab. Eng. Syst. Saf..

[17]  Alessandra Bagnato,et al.  Attribute Decoration of Attack-Defense Trees , 2012, Int. J. Secur. Softw. Eng..

[18]  Parvaiz Ahmed Khand.  System level security modeling using attack trees , 2009, 2009 2nd International Conference on Computer, Control and Communication.

[19]  Patrick Heymans,et al.  Visual syntax does matter: improving the cognitive effectiveness of the i* visual notation , 2010, Requirements Engineering.

[20]  Sushil Jajodia,et al.  Advanced Cyber Attack Modeling Analysis and Visualization , 2010 .

[21]  Chen-Yang Cheng,et al.  Application of fault tree analysis to assess inventory risk: a practical case from aerospace manufacturing , 2013 .

[22]  Yuji Yamaoka,et al.  Threat Tree Templates to Ease Difficulties in Threat Modeling , 2011, 2011 14th International Conference on Network-Based Information Systems.

[23]  W. Timothy Strayer,et al.  Using Machine Learning Techniques to Identify Botnet Traffic , 2006 .

[24]  Lane Harrison,et al.  Visualization evaluation for cyber security: trends and future directions , 2014, VizSEC.

[25]  R.F. Mills,et al.  Using Attack and Protection Trees to Analyze Threats and Defenses to Homeland Security , 2006, MILCOM 2006 - 2006 IEEE Military Communications conference.

[26]  Robert J. Ellison,et al.  Attack Trees , 2009, Encyclopedia of Biometrics.

[27]  Jeffrey L. Hieb,et al.  Cyber security risk assessment for SCADA and DCS networks. , 2007, ISA transactions.

[28]  Stefano Bistarelli,et al.  Defense trees for economic evaluation of security investments , 2006, First International Conference on Availability, Reliability and Security (ARES'06).

[29]  Peng Ning,et al.  Analyzing Intensive Intrusion Alerts via Correlation , 2002, RAID.

[30]  Bekir Sahin,et al.  Fault Tree Analysis of chemical cargo contamination by using fuzzy approach , 2015, Expert Syst. Appl..

[31]  Sushil Jajodia,et al.  Interactive Analysis of Attack Graphs Using Relational Queries , 2006, DBSec.

[32]  Alice M. Agogino,et al.  IDES: influence diagram based expert system , 1987 .

[33]  Ulrik Franke,et al.  Cyber Situational Awareness Testing , 2016 .

[34]  Sushil Jajodia,et al.  Efficient minimum-cost network hardening via exploit dependency graphs , 2003, 19th Annual Computer Security Applications Conference, 2003. Proceedings..

[35]  Xinming Ou,et al.  Googling Attack Graphs , 2007 .

[36]  Marc Dacier,et al.  Models and tools for quantitative assessment of operational security , 1996, SEC.

[37]  Ross D. Shachter.  Evaluating Influence Diagrams , 1986, Oper. Res..

[38]  Rodolphe Ortalo,et al.  Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security , 1999, IEEE Trans. Software Eng..

[39]  M. J. Reed,et al.  Attack graphs representations , 2012, 2012 4th Computer Science and Electronic Engineering Conference (CEEC).

[40]  Sergio Caltagirone,et al.  The Diamond Model of Intrusion Analysis , 2013 .

[41]  Faeiz Alserhani.  Knowledge-Based Model to Represent Security Information and Reason About Multi-stage Attacks , 2015, CAiSE Workshops.

[42]  S. Bhattacharya,et al.  A scalable representation towards attack graph generation , 2008, 2008 1st International Conference on Information Technology.

[43]  Chandan Mazumdar,et al.  A Graph Data Model for Attack Graph Generation and Analysis , 2014, SNDS.

[44]  Soumya K. Ghosh,et al.  A planner-based approach to generate and analyze minimal attack graph , 2010, Applied Intelligence.

[45]  Makis Stamatelatos,et al.  Fault tree handbook with aerospace applications , 2002 .

[46]  Edward G. Amoroso,et al.  Fundamentals of computer security technology , 1994 .

[47]  Eugene H. Spafford,et al.  ADEPTS: adaptive intrusion response using attack graphs in an e-commerce environment , 2005, 2005 International Conference on Dependable Systems and Networks (DSN'05).

[48]  Salvatore J. Bavuso,et al.  Dynamic fault-tree models for fault-tolerant computer systems , 1992 .

[49]  Youki Kadobayashi,et al.  Towards a Vulnerability Tree Security Evaluation of OpenStack's Logical Architecture , 2014, TRUST.

[50]  Marc Dacier,et al.  Privilege Graph: an Extension to the Typed Access Matrix Model , 1994, ESORICS.

[51]  Maybin K. Muyeba,et al.  Threat Modeling Revisited: Improving Expressiveness of Attack , 2008, 2008 Second UKSIM European Symposium on Computer Modeling and Simulation.

[52]  Mathias Ekstedt,et al.  Enterprise architecture models for cyber security analysis , 2009, 2009 IEEE/PES Power Systems Conference and Exposition.

[53]  Igor V. Kotenko,et al.  Attack Graph Based Evaluation of Network Security , 2006, Communications and Multimedia Security.

[54]  Somesh Jha,et al.  Two formal analyses of attack graphs , 2002, Proceedings 15th IEEE Computer Security Foundations Workshop. CSFW-15.

[55]  Peng Ning,et al.  Techniques and tools for analyzing intrusion alerts , 2004, TSEC.

[56]  Ping Wang,et al.  Threat risk analysis for cloud security based on Attack-Defense Trees , 2012, 2012 8th International Conference on Computing Technology and Information Management (NCM and ICNIT).

[57]  Jeffrey P. Landry,et al.  A Threat Tree for Health Information Security and Privacy , 2011, AMCIS.

[58]  Ross D. Shachter.  DAVID: influence diagram processing system for the macintosh , 1986, UAI.

[59]  Rayford B. Vaughn,et al.  An Approach to Model Network Exploitations Using Exploitation Graphs , 2006, Simul..

[60]  Sushil Jajodia,et al.  Automated Cyber Situation Awareness Tools and Models for Improving Analyst Performance , 2014, Cybersecurity Systems for Human Cognition Augmentation.

[61]  Ulf Lindqvist,et al.  Modeling multistep cyber attacks for scenario recognition , 2003, Proceedings DARPA Information Survivability Conference and Exposition.

[62]  George P. Tadda,et al.  Overview of Cyber Situation Awareness , 2010, Cyber Situational Awareness.

[63]  Xinming Ou,et al.  Improving Attack Graph Visualization through Data Reduction and Attack Grouping , 2008, VizSEC.

[64]  François-Xavier Aguessy,et al.  Dynamic Risk Assessment and Response Computation using Bayesian Attack Models. (Évaluation Dynamique de Risque et Calcul de Réponses Basés sur des Modèles d'Attaques Bayésiens) , 2016 .

[65]  Norman M. Sadeh,et al.  Learning to detect phishing emails , 2007, WWW '07.

[66]  James P. McDermott,et al.  Attack net penetration testing , 2001, NSPW '00.

[67]  Frédéric Cuppens,et al.  Alert correlation in a cooperative intrusion detection framework , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[68]  Jerald Dawkins,et al.  A structural framework for modeling multi-stage network attacks , 2002, Proceedings. International Conference on Parallel Processing Workshop.

[69]  Wenke Lee,et al.  Attack plan recognition and prediction using causal networks , 2004, 20th Annual Computer Security Applications Conference.

[70]  Mingye Liu,et al.  Fault Tree Analysis for Safety/Security Verification in Aviation Software , 2013 .

[71]  Sushil Jajodia,et al.  Correlating intrusion events and building attack scenarios through attack graph distances , 2004, 20th Annual Computer Security Applications Conference.

[72]  Johnnes Arreymbi,et al.  An examination of the security implications of the supervisory control and data acquisition (SCADA) system in a mobile networked environment: An augmented vulnerability tree approach. , 2010 .

[73]  Vasant Honavar,et al.  A Software Fault Tree Approach to Requirements Analysis of an Intrusion Detection System , 2002, Requirements Engineering.

[74]  Zied Elouedi,et al.  Naive Bayes vs decision trees in intrusion detection systems , 2004, SAC '04.

[75]  James L. Peterson,et al.  Petri Nets , 1977, CSUR.

[76]  Mathias Ekstedt,et al.  Cyber Security Risks Assessment with Bayesian Defense Graphs and Architectural Models , 2009 .

[77]  S. Vidalis,et al.  Using Vulnerability Trees for Decision Making in Threat Assessment , 2003 .

[78]  Benjamin S. Bloom,et al.  Handbook On Formative and Summative Evaluation of Student Learning , 1971 .

[79]  Sushil Jajodia,et al.  Minimum-cost network hardening using attack graphs , 2006, Comput. Commun..

[80]  Xia Wang,et al.  Software fault tree and coloured Petri net-based specification, design and implementation of agent-based intrusion detection systems , 2007, Int. J. Inf. Comput. Secur..

[81]  Felician Campean,et al.  A Function Failure Approach to Fault Tree Analysis for Automotive Systems , 2008 .

[82]  Markus Schumacher,et al.  Collaborative attack modeling , 2002, SAC '02.

[83]  Robert Lagerström,et al.  Extended Influence Diagram Generation , 2007, IESA.

[84]  Christoph Meinel,et al.  An Integrated Network Scanning Tool for Attack Graph Construction , 2011, GPC.

[85]  Kishor S. Trivedi,et al.  Performance and Reliability Analysis of Computer Systems: An Example-Based Approach Using the SHARPE Software Package , 2012 .

[86]  Fangcheng Tang,et al.  A Protection Tree Scheme for First-Failure Protection and Second-Failure Restoration in Optical Networks , 2005, ICCNMC.

[87]  David Coppit,et al.  Combining various solution techniques for dynamic fault tree analysis of computer systems , 1998, Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No.98EX231).

[88]  Shanchieh Jay Yang,et al.  Virtual terrain: a security-based representation of a computer network , 2008, SPIE Defense + Commercial Sensing.

[89]  Howard E. Lambert,et al.  Use of Fault Tree Analysis for Automotive Reliability and Safety Analysis , 2004 .

[90]  Xinming Ou,et al.  Practical IDS alert correlation in the face of dynamic threats , 2011 .

[91]  Andreas L. Opdahl,et al.  Experimental comparison of attack trees and misuse cases for security threat identification , 2009, Inf. Softw. Technol..