SoK: A Framework for Asset Discovery: Systematizing Advances in Network Measurements for Protecting Organizations

Asset discovery is fundamental to any organization's cybersecurity efforts. Indeed, one must accurately know which assets belong to an IT infrastructure before the infrastructure can be secured. While practitioners typically rely on a relatively small set of well-known techniques, the academic literature on the subject is voluminous. In particular, the Internet measurement research community has devised a number of asset discovery techniques to support many measurement studies over the past five years. In this paper, we systematize asset discovery techniques by constructing a framework that comprehensively captures how network identifiers and services are found. We extract asset discovery techniques from recent academic literature in security and networking and place them into the systematized framework. We then demonstrate how to apply the framework to several case studies of asset discovery workflows, which could aid research reproducibility. These case studies further suggest opportunities for researchers and practitioners to uncover and identify more assets than might be possible with traditional techniques.

[1]  Matthew P. Barrett,et al.  Framework for Improving Critical Infrastructure Cybersecurity, Version 1.1 (Arabic translation) , 2018 .

[2]  Deepak Kumar,et al.  Tracking Certificate Misissuance in the Wild , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[3]  Paul F. Syverson,et al.  Locating hidden servers , 2006, 2006 IEEE Symposium on Security and Privacy (S&P'06).

[4]  Taejoong Chung,et al.  Tunneling for Transparency: A Large-Scale Analysis of End-to-End Violations in the Internet , 2016, Internet Measurement Conference.

[5]  Jan Rüth,et al.  A First Look at QUIC in the Wild , 2018, PAM.

[6]  Walter Willinger,et al.  How Cloud Traffic Goes Hiding: A Study of Amazon's Peering Fabric , 2019, Internet Measurement Conference.

[7]  Nick Mathewson,et al.  Tor: The Second-Generation Onion Router , 2004, USENIX Security Symposium.

[8]  Adrienne Porter Felt,et al.  Measuring HTTPS Adoption on the Web , 2017, USENIX Security Symposium.

[9]  Vern Paxson,et al.  Target generation for internet-wide IPv6 scanning , 2017, Internet Measurement Conference.

[10]  Georg Carle,et al.  Scanning the IPv6 Internet: Towards a Comprehensive Hitlist , 2016, TMA.

[11]  Adam J. Aviv,et al.  Timing-based reconnaissance and defense in software-defined networks , 2016, ACSAC.

[12]  Bruce M. Maggs,et al.  Measuring and Applying Invalid SSL Certificates: The Silent Majority , 2016, Internet Measurement Conference.

[13]  Gorka Irazoqui Apecechea,et al.  Efficient, adversarial neighbor discovery using logical channels on Microsoft Azure , 2016, ACSAC.

[14]  Gang Wang,et al.  End-to-End Measurements of Email Spoofing Attacks , 2018, USENIX Security Symposium.

[15]  Lawrence K. Saul,et al.  Who is .com?: Learning to Parse WHOIS Records , 2015, Internet Measurement Conference.

[16]  Mark Allman,et al.  Don't Forget to Lock the Back Door! A Characterization of IPv6 Network Security Policy , 2016, NDSS.

[17]  Wouter Joosen,et al.  Herding Vulnerable Cats: A Statistical Approach to Disentangle Joint Responsibility for Web Security in Shared Hosting , 2017, CCS.

[18]  Andrew J. Kaizer,et al.  ~Open Resolvers: Understanding the Origins of Anomalous Open DNS Resolvers , 2015, PAM.

[19]  Christopher Rentrop,et al.  Shadow IT - Management and Control of Unofficial IT , 2012, ICDS 2012.

[20]  Alex Biryukov,et al.  Trawling for Tor Hidden Services: Detection, Measurement, Deanonymization , 2013, 2013 IEEE Symposium on Security and Privacy.

[21]  Christopher Krügel,et al.  Cloud Strife: Mitigating the Security Risks of Domain-Validated Certificates , 2018, NDSS.

[22]  Sharon Goldberg,et al.  The Unintended Consequences of Email Spam Prevention , 2018, PAM.

[23]  Aiko Pras,et al.  Measuring the Adoption of DDoS Protection Services , 2016, Internet Measurement Conference.

[24]  Mark Allman,et al.  On measuring the client-side DNS infrastructure , 2013, Internet Measurement Conference.

[25]  Juan Caballero,et al.  RevProbe: detecting silent reverse proxies in malicious server infrastructures , 2016, ACSAC.

[26]  Giovane C. M. Moura,et al.  Dmap: Automating Domain Name Ecosystem Measurements and Applications , 2018, 2018 Network Traffic Measurement and Analysis Conference (TMA).

[27]  Juan Caballero,et al.  CARONTE: Detecting Location Leaks for Deanonymizing Tor Hidden Services , 2015, CCS.

[28]  Parinaz Naghizadeh Ardabili,et al.  Cloudy with a Chance of Breach: Forecasting Cyber Security Incidents , 2015, USENIX Security Symposium.

[29]  Nicolas Christin,et al.  Measuring the Longitudinal Evolution of the Online Anonymous Marketplace Ecosystem , 2015, USENIX Security Symposium.

[30]  Aziz Mohaisen,et al.  Where Are You Taking Me? Behavioral Analysis of Open DNS Resolvers , 2019, 2019 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[31]  Matthew Green,et al.  Imperfect Forward Secrecy: How Diffie-Hellman Fails in Practice , 2015, CCS.

[32]  Stefan Savage,et al.  Security by Any Other Name: On the Effectiveness of Provider Based Email Security , 2015, CCS.

[33]  Konstantina Papagiannaki,et al.  Is the Web HTTP/2 Yet? , 2016, PAM.

[34]  Stephane Bortzmeyer DNS Query Name Minimisation to Improve Privacy , 2016, RFC.

[35]  Daiping Liu,et al.  All Your DNS Records Point to Us: Understanding the Security Threats of Dangling DNS Records , 2016, CCS.

[36]  Aiko Pras,et al.  On the Potential of IPv6 Open Resolvers for DDoS Attacks , 2017, PAM.

[37]  Han Zhang,et al.  A look at router geolocation in public and commercial databases , 2017, Internet Measurement Conference.

[38]  Yizheng Chen,et al.  Enabling Network Security Through Active DNS Datasets , 2016, RAID.

[39]  Christian Doerr,et al.  Just the Tip of the Iceberg: Internet-Scale Exploitation of Routers for Cryptojacking , 2019, CCS.

[40]  Robert Beverly,et al.  Server Siblings: Identifying Shared IPv4/IPv6 Infrastructure Via Active Fingerprinting , 2015, PAM.

[41]  Vaibhav Bajpai,et al.  Inferring persistent interdomain congestion , 2018, SIGCOMM.

[42]  Bram Klievink,et al.  Plug and Prey? Measuring the Commoditization of Cybercrime via Online Anonymous Markets , 2018, USENIX Security Symposium.

[43]  J. Alex Halderman,et al.  Illuminating the Security Issues Surrounding Lights-Out Server Management , 2013, WOOT.

[44]  Mingyan Liu,et al.  On the Mismanagement and Maliciousness of Networks , 2014, NDSS.

[45]  Christopher Krügel,et al.  Something from Nothing (There): Collecting Global IPv6 Datasets from DNS , 2017, PAM.

[46]  Leyla Bilge,et al.  Lean On Me: Mining Internet Service Dependencies From Large-Scale DNS Data , 2017, ACSAC.

[47]  Alberto Dainotti,et al.  Leveraging Internet Background Radiation for Opportunistic Network Analysis , 2015, Internet Measurement Conference.

[48]  J. Alex Halderman,et al.  A Search Engine Backed by Internet-Wide Scanning , 2015, CCS.

[49]  Roland van Rijswijk-Deij,et al.  TXTing 101: Finding Security Issues in the Long Tail of DNS TXT Records , 2020, 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW).

[50]  Benoit Donnet,et al.  Revisiting Subnet Inference WISE-ly , 2019, 2019 Network Traffic Measurement and Analysis Conference (TMA).

[51]  Marc Dacier,et al.  Circuit Fingerprinting Attacks: Passive Deanonymization of Tor Hidden Services , 2015, USENIX Security Symposium.

[52]  Ying Liu,et al.  A Reexamination of Internationalized Domain Names: The Good, the Bad and the Ugly , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[53]  Stefan Savage,et al.  You've Got Vulnerability: Exploring Effective Vulnerability Notifications , 2016, USENIX Security Symposium.

[54]  Zhou Li,et al.  Don't Let One Rotten Apple Spoil the Whole Barrel: Towards Automated Detection of Shadowed Domains , 2017, CCS.

[55]  J. Alex Halderman,et al.  FTP: The Forgotten Cloud , 2016, 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[56]  Giovane C. M. Moura,et al.  DNS Observatory: The Big Picture of the DNS , 2019, Internet Measurement Conference.

[57]  Georg Carle,et al.  The Rise of Certificate Transparency and Its Implications on the Internet Ecosystem , 2018, Internet Measurement Conference.

[58]  Gordon Fyodor Lyon,et al.  Nmap Network Scanning: The Official Nmap Project Guide to Network Discovery and Security Scanning , 2009 .

[59]  Ruian Duan,et al.  The Betrayal At Cloud City: An Empirical Analysis Of Cloud-Based Mobile Backends , 2019, USENIX Security Symposium.

[60]  Ying Liu,et al.  An End-to-End, Large-Scale Measurement of DNS-over-Encryption: How Far Have We Come? , 2019, Internet Measurement Conference.

[61]  Fengyuan Xu,et al.  Internet Protocol Cameras with No Password Protection: An Empirical Investigation , 2018, PAM.

[62]  Stefano Zanero,et al.  There's a Hole in that Bucket!: A Large-scale Analysis of Misconfigured S3 Buckets , 2018, ACSAC.

[63]  Anja Feldmann,et al.  BGP Prefix Delegations: A Deep Dive , 2016, Internet Measurement Conference.

[64]  Tsutomu Matsumoto,et al.  Detect Me If You… Oh Wait. An Internet-Wide View of Self-Revealing Honeypots , 2019, 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM).

[65]  Nicolas Christin,et al.  Traveling the silk road: a measurement analysis of a large anonymous online marketplace , 2012, WWW.

[66]  Tadayoshi Kohno,et al.  Satellite: Joint Analysis of CDNs and Network-Level Interference , 2016, USENIX Annual Technical Conference.

[67]  C. Kaufman Internet Key Exchange (IKEv2) Protocol", RFC 4306 , 2005 .

[68]  J. Alex Halderman,et al.  Measuring the Security Harm of TLS Crypto Shortcuts , 2016, Internet Measurement Conference.

[69]  Dave Levin,et al.  Residential links under the weather , 2019, SIGCOMM.

[70]  Haya Shulman,et al.  One Key to Sign Them All Considered Vulnerable: Evaluation of DNSSEC in the Internet , 2017, NSDI.

[71]  Jennifer Rexford,et al.  Bamboozling Certificate Authorities with BGP , 2018, USENIX Security Symposium.

[72]  Georg Carle,et al.  Large-scale classification of IPv6-IPv4 siblings with variable clock skew , 2016, 2017 Network Traffic Measurement and Analysis Conference (TMA).

[73]  Niklas Carlsson,et al.  Server-Side Adoption of Certificate Transparency , 2018, PAM.

[74]  Haya Shulman,et al.  Counting in the Dark: DNS Caches Discovery and Enumeration in the Internet , 2017, 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[75]  Davide Balzarotti,et al.  Uses and Abuses of Server-Side Requests , 2016, RAID.

[76]  Kyle Schomp,et al.  Characterization of Collaborative Resolution in Recursive DNS Resolvers , 2018, PAM.

[77]  Maciej Korczynski,et al.  Apples, oranges and hosting providers: Heterogeneity and security in the hosting market , 2016, NOMS 2016 - 2016 IEEE/IFIP Network Operations and Management Symposium.

[78]  J. Alex Halderman,et al.  Quack: Scalable Remote Measurement of Application-Layer Censorship , 2018, USENIX Security Symposium.

[79]  Wouter Joosen,et al.  Maneuvering Around Clouds: Bypassing Cloud-based Security Providers , 2015, CCS.

[80]  Patrick Sattler,et al.  Prefix Top Lists: Gaining Insights with Prefixes from Domain-based Top Lists on DNS Deployment , 2019, Internet Measurement Conference.

[81]  Angelique Faye Loe,et al.  You Shall Not Join: A Measurement Study of Cryptocurrency Peer-to-Peer Bootstrapping Techniques , 2019, CCS.

[82]  J. Alex Halderman,et al.  Neither Snow Nor Rain Nor MITM...: An Empirical Analysis of Email Delivery Security , 2015, Internet Measurement Conference.

[83]  Mitsuaki Akiyama,et al.  DomainScouter: Understanding the Risks of Deceptive IDNs , 2019, RAID.

[84]  Nicolas Christin,et al.  Three Case Studies in Quantitative Information Risk Analysis , 2008 .

[85]  Michael Rabinovich,et al.  A Look at the ECS Behavior of DNS Resolvers , 2019, Internet Measurement Conference.

[86]  Chase Cotton,et al.  Your Remnant Tells Secret: Residual Resolution in DDoS Protection Services , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[87]  Bruce M. Maggs,et al.  Measurement and Analysis of Private Key Sharing in the HTTPS Ecosystem , 2016, CCS.

[88]  Feng Qian,et al.  Under the Shadow of Sunshine: Understanding and Detecting Bulletproof Hosting on Legitimate Service Provider Networks , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[89]  Dave Levin,et al.  UAv6: Alias Resolution in IPv6 Using Unused Addresses , 2015, PAM.

[90]  Izzat Alsmadi,et al.  IoT and the Risk of Internet Exposure: Risk Assessment Using Shodan Queries , 2019, 2019 IEEE 20th International Symposium on "A World of Wireless, Mobile and Multimedia Networks" (WoWMoM).

[91]  Kimberly C. Claffy,et al.  Reasons Dynamic Addresses Change , 2016, Internet Measurement Conference.

[92]  Giovanni Vigna,et al.  Enumerating Active IPv6 Hosts for Large-Scale Security Scans via DNSSEC-Signed Reverse Zones , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[93]  Katharina Krombholz,et al.  Investigating System Operators' Perspective on Security Misconfigurations , 2018, CCS.

[94]  Aiko Pras,et al.  Passive Observations of a Large DNS Service: 2.5 Years in the Life of Google , 2018, 2018 Network Traffic Measurement and Analysis Conference (TMA).

[95]  Angelos Stavrou,et al.  End-Users Get Maneuvered: Empirical Analysis of Redirection Hijacking in Content Delivery Networks , 2018, USENIX Security Symposium.

[96]  Amir Herzberg,et al.  Practical Experience: Methodologies for Measuring Route Origin Validation , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[97]  Mitsuaki Akiyama,et al.  DomainProfiler: Discovering Domain Names Abused in Future , 2016, 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).

[98]  Alberto Dainotti,et al.  Millions of targets under attack: a macroscopic characterization of the DoS ecosystem , 2017, Internet Measurement Conference.

[99]  Max Mühlhäuser,et al.  Next Generation P2P Botnets: Monitoring Under Adverse Conditions , 2018, RAID.

[100]  Muhammad Rizwan Asghar,et al.  Measuring IPv6 DNS Reconnaissance Attacks and Preventing Them Using DNS Guard , 2018, 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN).