Are We Missing Labels? A Study of the Availability of Ground-Truth in Network Security Research

Network security is a long-lasting field of research constantly encountering new challenges. Inherently, research in this field is highly data-driven. Specifically, many approaches employ a supervised machine learning approach requiring labelled input data. While different publicly available data sets exist, labelling information is sparse. In order to understand how our community deals with this lack of labels, we perform a systematic study of network security research accepted at top IT security conferences in 2009-2013. Our analysis reveals that 70% of the papers reviewed rely on manually compiled data sets. Furthermore, only 10% of the studied papers release the data sets after compilation. This manifests that our community is facing a missing labelled data problem. In order to be able to address this problem, we give a definition and discuss crucial characteristics of the problem. Furthermore, we reflect and discuss roads towards overcoming this problem by establishing ground-truth and fostering data sharing.

[1]  Charles V. Wright,et al.  Generating Client Workloads and High-Fidelity Network Traffic for Controllable, Repeatable Experiments in Computer Security , 2010, RAID.

[2]  Yin Zhang,et al.  ViceROI: catching click-spam in search ad networks , 2013, CCS.

[3]  Fang Yu,et al.  Populated IP addresses: classification and applications , 2012, CCS.

[4]  Jonathon T. Giffin,et al.  Automatic Reverse Engineering of Malware Emulators , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[5]  Evangelos P. Markatos,et al.  A Generic Anonymization Framework for Network Traffic , 2006, 2006 IEEE International Conference on Communications.

[6]  Philip K. Chan,et al.  An Analysis of the 1999 DARPA/Lincoln Laboratory Evaluation Data for Network Anomaly Detection , 2003, RAID.

[7]  Leyla Bilge,et al.  EXPOSURE: Finding Malicious Domains Using Passive DNS Analysis , 2011, NDSS.

[8]  V. N. Venkatakrishnan,et al.  Blueprint: Robust Prevention of Cross-site Scripting Attacks for Existing Browsers , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[9]  Christopher Krügel,et al.  Clickonomics: Determining the Effect of Anti-Piracy Measures for One-Click Hosting , 2013, NDSS.

[10]  Tzi-cker Chiueh,et al.  Automatic Generation of String Signatures for Malware Detection , 2009, RAID.

[11]  Benjamin Livshits,et al.  Rozzle: De-cloaking Internet Malware , 2012, 2012 IEEE Symposium on Security and Privacy.

[12]  Kimberly C. Claffy,et al.  Dialing Privacy and Utility: A Proposed Data-Sharing Framework to Advance Internet Research , 2010, IEEE Security & Privacy.

[13]  A. Varga,et al.  THE OMNET++ DISCRETE EVENT SIMULATION SYSTEM , 2003 .

[14]  Peng Li,et al.  On Challenges in Evaluating Malware Clustering , 2010, RAID.

[15]  31st IEEE Symposium on Security and Privacy, S&P 2010, 16-19 May 2010, Berleley/Oakland, California, USA , 2010, IEEE Symposium on Security and Privacy.

[16]  Helen J. Wang,et al.  On the Incoherencies in Web Browser Access Control Policies , 2010, 2010 IEEE Symposium on Security and Privacy.

[17]  Somesh Jha,et al.  Recent Advances in Intrusion Detection, 13th International Symposium, RAID 2010, Ottawa, Ontario, Canada, September 15-17, 2010. Proceedings , 2010, RAID.

[18]  Farnam Jahanian,et al.  Improving Spam Blacklisting Through Dynamic Thresholding and Speculative Aggregation , 2010, NDSS.

[19]  Felix C. Freiling,et al.  Toward Automated Dynamic Malware Analysis Using CWSandbox , 2007, IEEE Secur. Priv..

[20]  Fang Yu,et al.  Investigation of Triangular Spamming: A Stealthy and Efficient Spamming Technique , 2010, 2010 IEEE Symposium on Security and Privacy.

[21]  Jun Sun,et al.  AUTHSCAN: Automatic Extraction of Web Authentication Protocols from Implementations , 2013, NDSS.

[22]  Robin Sommer,et al.  A Lone Wolf No More: Supporting Network Intrusion Detection with Real-Time Intelligence , 2012, RAID.

[23]  Falko Dressler,et al.  Dialog-based payload aggregation for intrusion detection , 2010, CCS '10.

[24]  Adam J. Aviv,et al.  Bridging the Data Gap: Data Related Challenges in Evaluating Large Scale Collaborative Security Systems , 2013, CSET.

[25]  M. Zimmer.  “But the data is already public”: on the ethics of research in Facebook , 2010, Ethics and Information Technology.

[26]  田端 利宏,et al.  Network and Distributed System Security Symposiumにおける研究動向の調査 , 2004 .

[27]  Mostafa H. Ammar,et al.  Prefix-preserving IP address anonymization: measurement-based security evaluation and a new cryptography-based scheme , 2004, Comput. Networks.

[28]  Salvatore J. Stolfo,et al.  Cross-Domain Collaborative Anomaly Detection: So Far Yet So Close , 2011, RAID.

[29]  Gianluca Stringhini,et al.  Shady paths: leveraging surfing crowds to detect malicious web pages , 2013, CCS.

[30]  Jun Li,et al.  Behavior-Based Worm Detectors Compared , 2010, RAID.

[31]  Stefan Savage,et al.  Juice: A Longitudinal Study of an SEO Botnet , 2013, NDSS.

[32]  Martina Lindorfer,et al.  Detecting Environment-Sensitive Malware , 2011, RAID.

[33]  Guofei Gu,et al.  Cross-Analysis of Botnet Victims: New Insights and Implications , 2011, RAID.

[34]  Aiko Pras,et al.  A Labeled Data Set for Flow-Based Intrusion Detection , 2009, IPOM.

[35]  Dawn Xiaodong Song,et al.  Dispatcher: enabling active botnet infiltration using automatic protocol reverse-engineering , 2009, CCS.

[36]  Irfan Ul Haq,et al.  What Is the Impact of P2P Traffic on Anomaly Detection? , 2010, RAID.

[37]  Aziz Mohaisen,et al.  Losing control of the internet: using the data plane to attack the control plane , 2010, CCS '10.

[38]  Salvatore J. Stolfo,et al.  Adaptive Anomaly Detection via Self-calibration and Dynamic Updating , 2009, RAID.

[39]  Carrie Gates,et al.  Coordinated Scan Detection , 2009, NDSS.

[40]  Vern Paxson,et al.  Outside the Closed World: On Using Machine Learning for Network Intrusion Detection , 2010, 2010 IEEE Symposium on Security and Privacy.

[41]  32nd IEEE Symposium on Security and Privacy, S&P 2011, 22-25 May 2011, Berkeley, California, USA , 2011, IEEE Symposium on Security and Privacy.

[42]  Alexander Aiken,et al.  Community Epidemic Detection Using Time-Correlated Anomalies , 2010, RAID.

[43]  Salvatore J. Stolfo,et al.  Research in attacks, intrusions, and defenses : 16th international symposium, RAID 2013, Rodney Bay, St. Lucia, October 23-25, 2013 : proceedings , 2013 .

[44]  Juan Caballero,et al.  FIRMA: Malware Clustering and Network Signature Generation with Mixed Network Behaviors , 2013, RAID.

[45]  Guanhua Yan,et al.  SMS-Watchdog: Profiling Social Behaviors of SMS Users for Anomaly Detection , 2009, RAID.

[46]  Fang Yu,et al.  Intention and Origination: An Inside Look at Large-Scale Bot Queries , 2013, NDSS.

[47]  Matthew Roughan,et al.  The need for simulation in evaluating anomaly detectors , 2008, CCRV.

[48]  Pierangela Samarati,et al.  Proceedings of the 8th ACM conference on Computer and Communications Security , 1998, CCS 2001.

[49]  Zihui Ge,et al.  ALERT-ID: Analyze Logs of the Network Element in Real Time for Intrusion Detection , 2012, RAID.

[50]  Thorsten Holz,et al.  IceShield: Detection and Mitigation of Malicious Websites with a Frozen DOM , 2011, RAID.

[51]  Christopher Krügel,et al.  Delta: automatic identification of unknown web-based infection campaigns , 2013, CCS.

[52]  Yingjie Zhou,et al.  Large-scale IP network behavior anomaly detection and identification using substructure-based approach and multivariate time series mining , 2012, Telecommun. Syst..

[53]  Christopher Krügel,et al.  Blacksheep: detecting compromised hosts in homogeneous crowds , 2012, CCS '12.

[54]  Christopher Krügel,et al.  Your botnet is my botnet: analysis of a botnet takeover , 2009, CCS.

[55]  John Heidemann,et al.  Uses and Challenges for Network Datasets , 2009, 2009 Cybersecurity Applications & Technology Conference for Homeland Security.

[56]  Malek Ben Salem,et al.  Modeling User Search Behavior for Masquerade Detection , 2011, RAID.

[57]  Feng Xiao,et al.  DSybil: Optimal Sybil-Resistance for Recommendation Systems , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[58]  David Brumley,et al.  BitShred: feature hashing malware for scalable triage and semantic analysis , 2011, CCS '11.

[59]  Christopher Krügel,et al.  Effective Anomaly Detection with Scarce Training Data , 2010, NDSS.

[60]  Salvatore J. Stolfo,et al.  Spectrogram: A Mixture-of-Markov-Chains Model for Anomaly Detection in Web Traffic , 2009, NDSS.

[61]  Somesh Jha,et al.  Recent advances in intrusion detection : 12th International Symposium, RAID 2009, Saint-Malo, France, September 23-25, 2009 : proceedings , 2009, RAID 2009.

[62]  Martín Abadi,et al.  Host Fingerprinting and Tracking on the Web: Privacy and Security Implications , 2012, NDSS.

[63]  Richard Lippmann,et al.  The 1999 DARPA off-line intrusion detection evaluation , 2000, Comput. Networks.

[64]  Nikita Borisov,et al.  RAINBOW: A Robust And Invisible Non-Blind Watermark for Network Flows , 2009, NDSS.

[65]  Sally Floyd,et al.  Difficulties in simulating the internet , 2001, TNET.

[66]  19th Annual Network and Distributed System Security Symposium, NDSS 2012, San Diego, California, USA, February 5-8, 2012 , 2012, NDSS.

[67]  R.K. Cunningham,et al.  Evaluating intrusion detection systems: the 1998 DARPA off-line intrusion detection evaluation , 2000, Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00.

[68]  Christopher Krügel,et al.  A Practical Attack to De-anonymize Social Network Users , 2010, 2010 IEEE Symposium on Security and Privacy.

[69]  Herbert Bos,et al.  Research in Attacks, Intrusions, and Defenses , 2015, Lecture Notes in Computer Science.

[70]  Chun-Ying Huang,et al.  Fast-Flux Bot Detection in Real Time , 2010, RAID.

[71]  Peng Ning,et al.  Authenticating Primary Users' Signals in Cognitive Radio Networks via Integrated Cryptographic and Wireless Link Signatures , 2010, 2010 IEEE Symposium on Security and Privacy.

[72]  Gregory J. Conti,et al.  Toward Instrumenting Network Warfare Competitions to Generate Labeled Datasets , 2009, CSET.

[73]  Jong Kim,et al.  WarningBird: Detecting Suspicious URLs in Twitter Stream , 2012, NDSS.

[74]  Abhinav Srivastava,et al.  Automatic Discovery of Parasitic Malware , 2010, RAID.

[75]  Zhou Li,et al.  Sidebuster: automated detection and quantification of side-channel leaks in web application development , 2010, CCS '10.

[76]  Christopher Krügel,et al.  Inspector Gadget: Automated Extraction of Proprietary Gadgets from Malware Binaries , 2010, 2010 IEEE Symposium on Security and Privacy.

[77]  Vern Paxson,et al.  Detecting stealthy, distributed SSH brute-forcing , 2013, CCS.

[78]  Xiapu Luo,et al.  A Centralized Monitoring Infrastructure for Improving DNS Security , 2010, RAID.

[79]  Charles V. Wright,et al.  Traffic Morphing: An Efficient Defense Against Statistical Traffic Analysis , 2009, NDSS.

[80]  Allison Bishop,et al.  Revocation Systems with Very Small Private Keys , 2010, 2010 IEEE Symposium on Security and Privacy.

[81]  Fang Yu,et al.  On Network-level Clusters for Spam Detection , 2010, NDSS.

[82]  Radu State,et al.  Proactive Discovery of Phishing Related Domain Names , 2012, RAID.

[83]  Felix C. Freiling,et al.  TrumanBox: Improving Dynamic Malware Analysis by Emulating the Internet , 2011, SSS.

[84]  Davide Balzarotti,et al.  Behind the Scenes of Online Attacks: an Analysis of Exploitation Behaviors on the Web , 2013, NDSS.

[85]  Scott E. Coull,et al.  On Measuring the Similarity of Network Hosts: Pitfalls, New Metrics, and Empirical Analyses , 2011, NDSS.

[86]  André Årnes,et al.  Anonymization of IP Traffic Monitoring Data: Attacks on Two Prefix-Preserving Anonymization Schemes and Some Proposed Remedies , 2005, Privacy Enhancing Technologies.

[87]  Thorsten Holz,et al.  Crouching tiger - hidden payload: security risks of scalable vectors graphics , 2011, CCS '11.

[88]  Chris Kanich,et al.  Botnet Judo: Fighting Spam with Itself , 2010, NDSS.

[89]  David Brumley,et al.  Automatically Inferring the Evolution of Malicious Activity on the Internet , 2013, NDSS.

[90]  Phillip A. Porras,et al.  Clear and Present Data: Opaque Traffic and its Security Implications for the Future , 2013, NDSS.

[91]  Pieter H. Hartel,et al.  Panacea: Automating Attack Classification for Anomaly-Based Network Intrusion Detection Systems , 2009, RAID.

[92]  Wenke Lee,et al.  The Core of the Matter: Analyzing Malicious Traffic in Cellular Carriers , 2013, NDSS.

[93]  Mostafa H. Ammar,et al.  On the design and performance of prefix-preserving IP traffic trace anonymization , 2001, IMW '01.

[94]  Sandro Etalle,et al.  N-Gram against the Machine: On the Feasibility of the N-Gram Network Analysis for Binary Protocols , 2012, RAID.

[95]  Wenke Lee,et al.  Connected Colors: Unveiling the Structure of Criminal Networks , 2013, RAID.

[96]  Salvatore J. Stolfo,et al.  Research in Attacks, Intrusions, and Defenses , 2013, Lecture Notes in Computer Science.

[97]  Somesh Jha,et al.  Synthesizing Near-Optimal Malware Specifications from Suspicious Behaviors , 2010, 2010 IEEE Symposium on Security and Privacy.

[98]  Kevin Borders,et al.  Quantifying Information Leaks in Outbound Web Traffic , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[99]  Michael Backes,et al.  Automatic Discovery and Quantification of Information Leaks , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[100]  Konstantina Papagiannaki,et al.  Exploiting Temporal Persistence to Detect Covert Botnet Channels , 2009, RAID.

[101]  Kang G. Shin,et al.  RB-Seeker: Auto-detection of Redirection Botnets , 2009, NDSS.

[102]  Vitaly Shmatikov,et al.  Large-scale collection and sanitization of network security data: risks and challenges , 2006, NSPW '06.

[103]  Fabian Monrose,et al.  Phonotactic Reconstruction of Encrypted VoIP Conversations: Hookt on Fon-iks , 2011, 2011 IEEE Symposium on Security and Privacy.

[104]  Adrian Perrig,et al.  Round-Efficient Broadcast Authentication Protocols for Fixed Topology Classes , 2010, 2010 IEEE Symposium on Security and Privacy.

[105]  Weijia Jia,et al.  A new cell counter based attack against tor , 2009, CCS.

[106]  Mabry Tyson,et al.  FRESCO: Modular Composable Security Services for Software-Defined Networks , 2013, NDSS.

[107]  Vinod Yegneswaran,et al.  AVANT-GUARD: scalable and vigilant switch flow management in software-defined networks , 2013, CCS.

[108]  Sungryoul Lee,et al.  Kargus: a highly-scalable software-based intrusion detection system , 2012, CCS.

[109]  Recent Advances in Intrusion Detection - 14th International Symposium, RAID 2011, Menlo Park, CA, USA, September 20-21, 2011. Proceedings , 2011, RAID.

[110]  Vern Paxson,et al.  Issues and etiquette concerning use of shared measurement data , 2007, IMC '07.

[111]  Wenke Lee,et al.  SURF: detecting and measuring search poisoning , 2011, CCS '11.

[112]  Christopher Krügel,et al.  Identifying Dormant Functionality in Malware Programs , 2010, 2010 IEEE Symposium on Security and Privacy.

[113]  Sotiris Ioannidis,et al.  MIDeA: a multi-parallel intrusion detection architecture , 2011, CCS '11.

[114]  He Liu,et al.  Click Trajectories: End-to-End Analysis of the Spam Value Chain , 2011, 2011 IEEE Symposium on Security and Privacy.

[115]  Thomas Engel,et al.  Runtime Monitoring and Dynamic Reconfiguration for Intrusion Detection Systems , 2009, RAID.

[116]  Christopher Krügel,et al.  BotFinder: finding bots in network traffic without deep packet inspection , 2012, CoNEXT '12.

[117]  Fang Yu,et al.  Knowing your enemy: understanding and detecting malicious web advertising , 2012, CCS '12.

[118]  Michalis Polychronakis,et al.  Server-Side Code Injection Attacks: A Historical Perspective , 2013, RAID.

[119]  Nan Jiang,et al.  Understanding SMS Spam in a Large Cellular Network: Characteristics, Strategies and Defenses , 2013, RAID.

[120]  Slobodan Petrovic,et al.  Passive Remote Source NAT Detection Using Behavior Statistics Derived from NetFlow , 2013, AIMS.

[121]  Wenke Lee,et al.  Beheading hydras: performing effective botnet takedowns , 2013, CCS.

[122]  Vitaly Shmatikov,et al.  The Parrot Is Dead: Observing Unobservable Network Communications , 2013, 2013 IEEE Symposium on Security and Privacy.

[123]  Ehab Al-Shaer,et al.  Proceedings of the 2009 ACM Conference on Computer and Communications Security, CCS 2009, Chicago, Illinois, USA, November 9-13, 2009 , 2009, CCS.

[124]  Hung Q. Ngo,et al.  A Data-Centric Approach to Insider Attack Detection in Database Systems , 2010, RAID.

[125]  Ting Yu,et al.  the ACM Conference on Computer and Communications Security, CCS'12, Raleigh, NC, USA, October 16-18, 2012 , 2012, CCS.

[126]  Simin Nadjm-Tehrani,et al.  Anomaly Detection and Mitigation for Disaster Area Networks , 2010, RAID.

[127]  Radu State,et al.  Automated Behavioral Fingerprinting , 2009, RAID.

[128]  John Riedl,et al.  You are what you say: privacy risks of public mentions , 2006, SIGIR '06.

[129]  Jason Lee,et al.  The devil and packet trace anonymization , 2006, CCRV.

[130]  John McHugh,et al.  Testing Intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by Lincoln Laboratory , 2000, TSEC.

[131]  Syed Ali Khayam,et al.  On achieving good operating points on an ROC plane using stochastic anomaly score prediction , 2009, CCS.

[132]  Guofei Gu,et al.  NEIGHBORWATCHER: A Content-Agnostic Comment Spam Inference System , 2013, NDSS.

[133]  Leyla Bilge,et al.  Disclosure: detecting botnet command and control servers through large-scale NetFlow analysis , 2012, ACSAC '12.

[134]  Hiroki Takakura,et al.  Statistical analysis of honeypot data and building of Kyoto 2006+ dataset for NIDS evaluation , 2011, BADGERS '11.

[135]  Paolo Milani Comparetti,et al.  EvilSeed: A Guided Approach to Finding Malicious Web Pages , 2012, 2012 IEEE Symposium on Security and Privacy.

[136]  Peng Li,et al.  Automatically Adapting a Trained Anomaly Detector to Software Patches , 2009, RAID.

[137]  Lei Liu,et al.  VirusMeter: Preventing Your Cellphone from Spies , 2009, RAID.

[138]  Ming Zhang,et al.  Pretty-Bad-Proxy: An Overlooked Adversary in Browsers' HTTPS Deployments , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[139]  Syed Ali Khayam,et al.  Revisiting Traffic Anomaly Detection Using Software Defined Networking , 2011, RAID.

[140]  Nikita Borisov,et al.  SWIRL: A Scalable Watermark to Detect Correlated Network Flows , 2011, NDSS.

[141]  Fang Yu,et al.  Finding the Linchpins of the Dark Web: a Study on Topologically Dedicated Hosts on Malicious Web Infrastructures , 2013, 2013 IEEE Symposium on Security and Privacy.

[142]  Herbert Bos,et al.  Prudent Practices for Designing Malware Experiments: Status Quo and Outlook , 2012, 2012 IEEE Symposium on Security and Privacy.

[143]  Engin Kirda,et al.  Automated Discovery of Parameter Pollution Vulnerabilities in Web Applications , 2011, NDSS.

[144]  30th IEEE Symposium on Security and Privacy (S&P 2009), 17-20 May 2009, Oakland, California, USA , 2009, IEEE Symposium on Security and Privacy.

[145]  Christopher Krügel,et al.  Prospex: Protocol Specification Extraction , 2009, 2009 30th IEEE Symposium on Security and Privacy.

[146]  Dawn Xiaodong Song,et al.  Design and Evaluation of a Real-Time URL Spam Filtering Service , 2011, 2011 IEEE Symposium on Security and Privacy.

[147]  Mitsuaki Akiyama,et al.  Active Credential Leakage for Observing Web-Based Attack Cycle , 2013, RAID.

[148]  Radu State,et al.  BotCloud: Detecting botnets using MapReduce , 2011, 2011 IEEE International Workshop on Information Forensics and Security.

[149]  Chao Yang,et al.  PoisonAmplifier: A Guided Approach of Discovering Compromised Websites through Reversing Search Poisoning Attacks , 2012, RAID.

[150]  Konrad Rieck,et al.  Deobfuscating Embedded Malware Using Probable-Plaintext Attacks , 2013, RAID.