A Survey on Security Metrics

The importance of security metrics can hardly be overstated. Despite the attention that has been paid by the academia, government and industry in the past decades, this important problem stubbornly remains open. In this survey, we present a survey of knowledge on security metrics. The survey is centered on a novel taxonomy, which classifies security metrics into four categories: metrics for measuring the system vulnerabilities, metrics for measuring the defenses, metrics for measuring the threats, and metrics for measuring the situations. The insight underlying the taxonomy is that situations (or outcomes of cyber attack-defense interactions) are caused by certain threats (or attacks) against systems that have certain vulnerabilities (including human factors) and employ certain defenses. In addition to systematically reviewing the security metrics that have been proposed in the literature, we discuss the gaps between the state of the art and the ultimate goals.

[1]  Michael Backes,et al.  Oxymoron: Making Fine-Grained Memory Randomization Practical by Allowing Code Sharing , 2014, USENIX Security Symposium.

[2]  Tudor Dumitras,et al.  Vulnerability Disclosure in the Age of Social Media: Exploiting Twitter for Predicting Real-World Exploits , 2015, USENIX Security Symposium.

[3]  Vern Paxson,et al.  When Governments Hack Opponents: A Look at Actors and Technology , 2014, USENIX Security Symposium.

[4]  Zinta S. Byrne,et al.  The Psychology of Security for the Home Computer User , 2012, 2012 IEEE Symposium on Security and Privacy.

[5]  Nick Feamster,et al.  ASwatch: An AS Reputation System to Expose Bulletproof Hosting ASes , 2015, SIGCOMM.

[6]  Shouhuai Xu,et al.  An evasion and counter-evasion study in malicious websites detection , 2014, 2014 IEEE Conference on Communications and Network Security.

[7]  Xinwen Zhang,et al.  After we knew it: empirical study and modeling of cost-effectiveness of exploiting prevalent known vulnerabilities across IaaS cloud , 2014, AsiaCCS.

[8]  Joshua Taylor,et al.  A Quantitative Framework for Moving Target Defense Effectiveness Evaluation , 2015, MTD@CCS.

[9]  Fabio Massacci,et al.  Comparing Vulnerability Severity and Exploits Using Case-Control Studies , 2014, TSEC.

[10]  Shouhuai Xu,et al.  Active cyber defense dynamics exhibiting rich phenomena , 2015, HotSoS.

[11]  Pavel Laskov,et al.  Practical Evasion of a Learning-Based Classifier: A Case Study , 2014, 2014 IEEE Symposium on Security and Privacy.

[12]  Fan Long,et al.  Control Jujutsu: On the Weaknesses of Fine-Grained Control Flow Integrity , 2015, CCS.

[13]  Amos Azaria,et al.  Behavioral Analysis of Insider Threat: A Survey and Bootstrapped Prediction in Imbalanced Data , 2014, IEEE Transactions on Computational Social Systems.

[14]  Somesh Jha,et al.  Automated generation and analysis of attack graphs , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[15]  David A. Schmidt,et al.  Aggregating vulnerability metrics in enterprise networks using attack graphs , 2013, J. Comput. Secur..

[16]  Luca Allodi,et al.  The Heavy Tails of Vulnerability Exploitation , 2015, ESSoS.

[17]  Michael K. Reiter,et al.  An Epidemiological Study of Malware Encounters in a Large Enterprise , 2014, CCS.

[18]  Shouhuai Xu,et al.  A Stochastic Model of Multivirus Dynamics , 2012, IEEE Transactions on Dependable and Secure Computing.

[19]  Christian Rossow,et al.  Paint It Black: Evaluating the Effectiveness of Malware Blacklists , 2014, RAID.

[20]  Samuel Kounev,et al.  Evaluating Computer Intrusion Detection Systems , 2015, ACM Comput. Surv..

[21]  Joseph Bonneau.  Statistical Metrics for Individual Password Strength , 2012, Security Protocols Workshop.

[22]  Shouhuai Xu,et al.  Emergent behavior in cybersecurity , 2014, HotSoS '14.

[23]  Butler W. Lampson.  Practical Principles for Computer Security , 2007 .

[24]  Mário S. Alvim,et al.  Quantifying Information Flow for Dynamic Secrets , 2014, 2014 IEEE Symposium on Security and Privacy.

[25]  Sudhir Aggarwal,et al.  Testing metrics for password creation policies by attacking large sets of revealed passwords , 2010, CCS '10.

[26]  John S. Baras,et al.  A framework for the evaluation of intrusion detection systems , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[27]  Wenke Lee,et al.  Modeling Botnet Propagation Using Time Zones , 2006, NDSS.

[28]  Paul Ammann,et al.  Using model checking to analyze network vulnerabilities , 2000, Proceeding 2000 IEEE Symposium on Security and Privacy. S&P 2000.

[29]  Ehab Al-Shaer,et al.  A comprehensive objective network security metric framework for proactive security configuration , 2008, CSIIRW '08.

[30]  Thomas C. Eskridge,et al.  VINE: A Cyber Emulation Environment for MTD Experimentation , 2015, MTD@CCS.

[31]  Bruce Schneier,et al.  Secrets and Lies: Digital Security in a Networked World , 2000 .

[32]  William Herlands,et al.  Effective Entropy: Security-Centric Metric for Memory Randomization Techniques , 2014, CSET.

[33]  Sushil Jajodia,et al.  A weakest-adversary security metric for network configuration security analysis , 2006, QoP '06.

[34]  Dan Boneh,et al.  CCFI: Cryptographically Enforced Control Flow Integrity , 2015, CCS.

[35]  Shouhuai Xu,et al.  Predicting Cyber Attack Rates With Extreme Values , 2015, IEEE Transactions on Information Forensics and Security.

[36]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[37]  Wayne A. Jansen,et al.  Directions in Security Metrics Research , 2009 .

[38]  Andreas Terzis,et al.  On the Effectiveness of Distributed Worm Monitoring , 2005, USENIX Security Symposium.

[39]  Anja Feldmann,et al.  An Assessment of Overt Malicious Activity Manifest in Residential Networks , 2011, DIMVA.

[40]  Per Larsen,et al.  SoK: Automated Software Diversity , 2014, 2014 IEEE Symposium on Security and Privacy.

[41]  Martin C. Libicki,et al.  Markets for Cybercrime Tools and Stolen Data: Hackers' Bazaar , 2014 .

[42]  Parinaz Naghizadeh Ardabili,et al.  Cloudy with a Chance of Breach: Forecasting Cyber Security Incidents , 2015, USENIX Security Symposium.

[43]  Mathias Payer,et al.  Control-Flow Integrity , 2017, ACM Comput. Surv..

[44]  Bharat B. Madan,et al.  Modeling and quantification of security attributes of software systems , 2002, Proceedings International Conference on Dependable Systems and Networks.

[45]  Scott A. DeLoach,et al.  Metrics of Security , 2014, Cyber Defense and Situational Awareness.

[46]  Ponnurangam Kumaraguru,et al.  Who falls for phish?: a demographic analysis of phishing susceptibility and effectiveness of interventions , 2010, CHI.

[47]  Shouhuai Xu,et al.  Adaptive Epidemic Dynamics in Networks , 2013, ACM Trans. Auton. Adapt. Syst..

[48]  Davide Balzarotti,et al.  SoK: Deep Packer Inspection: A Longitudinal Study of the Complexity of Run-Time Packers , 2015, 2015 IEEE Symposium on Security and Privacy.

[49]  Yvo Desmedt,et al.  Threshold Cryptosystems , 1989, CRYPTO.

[50]  Andrew Jaquith.  Security Metrics: Replacing Fear, Uncertainty, and Doubt , 2007 .

[51]  David A. Wagner,et al.  ROP is Still Dangerous: Breaking Modern Defenses , 2014, USENIX Security Symposium.

[52]  J. Doug Tygar,et al.  Adversarial machine learning , 2019, AISec '11.

[53]  David Brumley,et al.  Automatic Patch-Based Exploit Generation is Possible: Techniques and Implications , 2008, 2008 IEEE Symposium on Security and Privacy (sp 2008).

[54]  Tomas Olovsson,et al.  A Quantitative Model of the Security Intrusion Process Based on Attacker Behavior , 1997, IEEE Trans. Software Eng..

[55]  Shouhuai Xu,et al.  A new approach to modeling and analyzing security of networked systems , 2014, HotSoS '14.

[56]  Ben Niu,et al.  Per-Input Control-Flow Integrity , 2015, CCS.

[57]  Cynthia Dwork,et al.  Differential Privacy , 2006, ICALP.

[58]  Guofei Gu,et al.  A Taxonomy of Botnet Structures , 2007, Twenty-Third Annual Computer Security Applications Conference (ACSAC 2007).

[59]  Nitesh Saxena,et al.  A Multi-Modal Neuro-Physiological Study of Phishing Detection and Malware Warnings , 2015, CCS.

[60]  Shouhuai Xu,et al.  A Characterization of Cybersecurity Posture from Network Telescope Data , 2014, INTRUST.

[61]  Barton P. Miller,et al.  Binary-code obfuscations in prevalent packer tools , 2013, CSUR.

[62]  G. Ramalingam,et al.  The undecidability of aliasing , 1994, TOPL.

[63]  Aziz Mohaisen,et al.  AV-Meter: An Evaluation of Antivirus Scans and Labels , 2014, DIMVA.

[64]  Ramesh Karri,et al.  A Primer on Hardware Security: Models, Methods, and Metrics , 2014, Proceedings of the IEEE.

[65]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[66]  D. Pinto.  Secrets and Lies: Digital Security in a Networked World , 2003 .

[67]  Leyla Bilge,et al.  Before we knew it: an empirical study of zero-day attacks in the real world , 2012, CCS.

[68]  Blase Ur,et al.  Measuring Real-World Accuracies and Biases in Modeling Password Guessability , 2015, USENIX Security Symposium.

[69]  Joseph Bonneau,et al.  The Science of Guessing: Analyzing an Anonymized Corpus of 70 Million Passwords , 2012, 2012 IEEE Symposium on Security and Privacy.

[70]  David Levin.  Lessons learned in using live red teams in IA experiments , 2003, Proceedings DARPA Information Survivability Conference and Exposition.

[71]  Ray A. Perlner,et al.  Electronic Authentication Guideline , 2014 .

[72]  Carl E. Landwehr,et al.  A taxonomy of computer program security flaws , 1993, CSUR.

[73]  Cynthia A. Phillips,et al.  A graph-based system for network-vulnerability analysis , 1998, NSPW '98.

[74]  Úlfar Erlingsson,et al.  Enforcing Forward-Edge Control-Flow Integrity in GCC & LLVM , 2014, USENIX Security Symposium.

[75]  Vern Paxson,et al.  The Matter of Heartbleed , 2014, Internet Measurement Conference.

[76]  Jean-Yves Le Boudec,et al.  Quantifying Location Privacy , 2011, 2011 IEEE Symposium on Security and Privacy.

[77]  Salvatore J. Stolfo,et al.  ALDR: A New Metric for Measuring Effective Layering of Defenses , 2011 .

[78]  Shouhuai Xu,et al.  A Stochastic Model of Active Cyber Defense Dynamics , 2015, Internet Math..

[79]  Guofei Gu,et al.  Measuring intrusion detection capability: an information-theoretic approach , 2006, ASIACCS '06.

[80]  Ahmad-Reza Sadeghi,et al.  Stitching the Gadgets: On the Ineffectiveness of Coarse-Grained Control-Flow Integrity Protection , 2014, USENIX Security Symposium.

[81]  Herbert Bos,et al.  Out of Control: Overcoming Control-Flow Integrity , 2014, 2014 IEEE Symposium on Security and Privacy.

[82]  Hovav Shacham,et al.  On the effectiveness of address-space randomization , 2004, CCS '04.

[83]  Shouhuai Xu,et al.  Characterizing Honeypot-Captured Cyber Attacks: Statistical Framework and Case Study , 2013, IEEE Transactions on Information Forensics and Security.

[84]  Duminda Wijesekera,et al.  Scalable, graph-based network vulnerability analysis , 2002, CCS '02.

[85]  Sonia Chiasson,et al.  A clinical study of risk factors related to malware infections , 2013, CCS.

[86]  Pedro M. Domingos,et al.  Adversarial classification , 2004, KDD.

[87]  Stefan Axelsson,et al.  The base-rate fallacy and its implications for the difficulty of intrusion detection , 1999, CCS '99.

[88]  Michael P. Wellman,et al.  Empirical Game-Theoretic Analysis for Moving Target Defense , 2015, MTD@CCS.

[89]  Chuanyi Ji,et al.  Measuring Network-Aware Worm Spreading Ability , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[90]  Shouhuai Xu,et al.  Cybersecurity dynamics , 2014, HotSoS '14.

[91]  John E. Gaffney,et al.  Evaluation of intrusion detectors: a decision theory approach , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[92]  Somesh Jha,et al.  Two formal analyses of attack graphs , 2002, Proceedings 15th IEEE Computer Security Foundations Workshop. CSFW-15.

[93]  Salvatore J. Stolfo,et al.  Measuring Drive-by Download Defense in Depth , 2014, RAID.

[94]  Shouhuai Xu,et al.  Cyber Epidemic Models with Dependences , 2015, Internet Math..

[95]  Wenke Lee,et al.  ASLR-Guard: Stopping Address Space Leakage for Code Reuse Attacks , 2015, CCS.

[96]  Sushil Jajodia,et al.  k-Zero Day Safety: Measuring the Security Risk of Networks against Unknown Attacks , 2010, ESORICS.

[97]  Ehab Al-Shaer,et al.  A Novel Quantitative Approach For Measuring Network Security , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[98]  Robert K. Cunningham,et al.  Why Measuring Security Is Hard , 2010, IEEE Security & Privacy.

[99]  William H. Sanders,et al.  Model-based Security Metrics Using ADversary VIew Security Evaluation (ADVISE) , 2011, 2011 Eighth International Conference on Quantitative Evaluation of SysTems.

[100]  Lujo Bauer,et al.  Guess Again (and Again and Again): Measuring Password Strength by Simulating Password-Cracking Algorithms , 2011, 2012 IEEE Symposium on Security and Privacy.

[101]  Tamara Yu,et al.  Continuous Security Metrics for Prevalent Network Threats: Introduction and First Four Metrics , 2012 .

[102]  Stephen H. Conrad,et al.  A behavioral theory of insider-threat risks: A system dynamics approach , 2008, TOMC.

[103]  Shari Lawrence Pfleeger.  Useful Cybersecurity Metrics , 2009, IT Professional.

[104]  Kevin M. Stine,et al.  Performance Measurement Guide for Information Security , 2008 .

[105]  Ahmad-Reza Sadeghi,et al.  Just-In-Time Code Reuse: On the Effectiveness of Fine-Grained Address Space Layout Randomization , 2013, 2013 IEEE Symposium on Security and Privacy.

[106]  Amir Moradi,et al.  Leakage Assessment Methodology - A Clear Roadmap for Side-Channel Evaluations , 2015, CHES.

[107]  Shouhuai Xu,et al.  An Extended Stochastic Model for Quantitative Security Analysis of Networked Systems , 2012, Internet Math..

[108]  Eric Wustrow,et al.  Mining Your Ps and Qs: Detection of Widespread Weak Keys in Network Devices , 2012, USENIX Security Symposium.

[109]  Kevin C. Almeroth,et al.  FIRE: FInding Rogue nEtworks , 2009, 2009 Annual Computer Security Applications Conference.

[110]  Mingyan Liu,et al.  On the Mismanagement and Maliciousness of Networks , 2014, NDSS.

[111]  Mihir Bellare,et al.  A concrete security treatment of symmetric encryption , 1997, Proceedings 38th Annual Symposium on Foundations of Computer Science.

[112]  Leyla Bilge,et al.  The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching , 2015, 2015 IEEE Symposium on Security and Privacy.

[113]  Rodolphe Ortalo,et al.  Experimenting with Quantitative Evaluation Tools for Monitoring Operational Security , 1999, IEEE Trans. Software Eng..

[114]  William H. Sanders.  Quantitative Security Metrics: Unattainable Holy Grail or a Vital Breakthrough within Our Reach? , 2014, IEEE Security & Privacy.

[115]  Geoffrey Thomas,et al.  Security Impact Ratings Considered Harmful , 2009, HotOS.

[116]  Jeannette M. Wing,et al.  An Attack Surface Metric , 2011, IEEE Transactions on Software Engineering.

[117]  Shouhuai Xu,et al.  Characterizing the power of moving target defense via cyber epidemic dynamics , 2014, HotSoS '14.

[118]  Dawn Xiaodong Song,et al.  SoK: Eternal War in Memory , 2013, 2013 IEEE Symposium on Security and Privacy.

[119]  Adam Senft,et al.  Targeted Threat Index: Characterizing and Quantifying Politically-Motivated Targeted Malware , 2014, USENIX Security Symposium.

[120]  David A. Wagner,et al.  Control-Flow Bending: On the Effectiveness of Control-Flow Integrity , 2015, USENIX Security Symposium.

[121]  Shouhuai Xu,et al.  A Stochastic Model for Quantitative Security Analyses of Networked Systems , 2016, IEEE Transactions on Dependable and Secure Computing.

[122]  Nicolas Christin,et al.  Metrics for Measuring ISP Badness: The Case of Spam - (Short Paper) , 2012, Financial Cryptography.

[123]  Tudor Dumitras,et al.  Some Vulnerabilities Are Different Than Others - Studying Vulnerabilities and Attack Surfaces in the Wild , 2014, RAID.

[124]  Marc Dacier,et al.  Models and tools for quantitative assessment of operational security , 1996, SEC.

[125]  Salvatore J. Stolfo,et al.  Heisenbyte: Thwarting Memory Disclosure Attacks using Destructive Code Reads , 2015, CCS.

[126]  Hovav Shacham,et al.  When private keys are public: results from the 2008 Debian OpenSSL vulnerability , 2009, IMC '09.

[127]  Hannes Holm.  A Large-Scale Study of the Time Required to Compromise a Computer System , 2014, IEEE Transactions on Dependable and Secure Computing.

[128]  Kathleen M. Carley,et al.  An empirical study of global malware encounters , 2015, HotSoS.

[129]  T. H. Bryant,et al.  SOCIETIES AND ACADEMIES. , 1895 .

[130]  Christopher Meek,et al.  Adversarial learning , 2005, KDD '05.

[131]  Sushil Jajodia,et al.  Time-efficient and cost-effective network hardening using attack graphs , 2012, IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012).

[132]  Shouhuai Xu.  Analyzing Malware Detection Efficiency with Multiple Anti-Malware Programs , 2012 .