Demystifying Internet-Wide Service Discovery

This paper develops a high-performance, Internet-wide service discovery tool, which we call IRLscanner, whose main design objectives have been to maximize politeness at remote networks, allow scanning rates that achieve coverage of the Internet in minutes/hours (rather than weeks/months), and significantly reduce administrator complaints. Using IRLscanner and 24-h scans, we perform 21 Internet-wide experiments using six different protocols (i.e., DNS, HTTP, SMTP, EPMAP, ICMP, and UDP ECHO), demonstrate the usefulness of ACK scans in detecting live hosts behind stateless firewalls, and undertake the first Internet-wide OS fingerprinting. In addition, we analyze the feedback generated (e.g., complaints, IDS alarms) and suggest novel approaches for reducing the amount of blowback during similar studies, which should enable researchers to collect valuable experimental data in the future with significantly fewer hurdles.

[1]  Internet Assigned Numbers Authority.  Special-Use IPv4 Addresses , 2002, RFC.

[2]  Stefan Savage,et al.  The Spread of the Sapphire/Slammer Worm , 2003 .

[3]  John S. Heidemann,et al.  Understanding passive and active service discovery , 2007, IMC '07.

[4]  Tal Garfinkel,et al.  Opportunistic Measurement: Extracting Insight from Spurious Traffic , 2005 .

[5]  Christos Gkantsidis,et al.  Sampling Strategies for Epidemic-Style Information Dissemination , 2008, INFOCOM 2008.

[6]  Don Towsley,et al.  Routing worm: a fast, selective attack worm based on IP address information , 2005, Workshop on Principles of Advanced and Distributed Simulation (PADS'05).

[7]  Vinod Yegneswaran,et al.  Characteristics of internet background radiation , 2004, IMC '04.

[8]  Phillip A. Porras,et al.  Highly Predictive Blacklisting , 2008, USENIX Security Symposium.

[9]  Yoichi Shinoda,et al.  Vulnerabilities of Passive Internet Threat Monitors , 2005, USENIX Security Symposium.

[10]  C. Lee Giles,et al.  Accessibility of information on the web , 1999, Nature.

[11]  Dmitri Loguinov,et al.  Modeling Heterogeneous User Churn and Local Resilience of Unstructured P2P Networks , 2006, Proceedings of the 2006 IEEE International Conference on Network Protocols.

[12]  Ramesh Govindan,et al.  Heuristics for Internet map discovery , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[13]  Donald Ervin Knuth,et al.  The Art of Computer Programming, 2nd Ed. (Addison-Wesley Series in Computer Science and Information , 1978 .

[14]  Vern Paxson,et al.  Automating analysis of large-scale botnet probing events , 2009, ASIACCS '09.

[15]  Niels Provos,et al.  Corrupted DNS Resolution Paths: The Rise of a Malicious Resolution Authority , 2008, NDSS.

[16]  Sally Floyd,et al.  Measuring the evolution of transport protocols in the internet , 2005, CCRV.

[17]  Ramesh Govindan,et al.  Census and survey of the visible internet , 2008, IMC '08.

[18]  Vern Paxson,et al.  A brief history of scanning , 2007, IMC '07.

[19]  Songwu Lu,et al.  IPv4 address allocation and the BGP routing table evolution , 2005, CCRV.

[20]  Sally Floyd,et al.  On inferring TCP behavior , 2001, SIGCOMM 2001.

[21]  Stuart McClure,et al.  Hacking Exposed: Network Security Secrets and Solutions, Fourth Edition , 2001 .

[22]  Dmitri Loguinov,et al.  Enabling High-Performance Internet-Wide Measurements on Windows , 2010, PAM.

[23]  Mark Allman,et al.  Estimating loss rates with TCP , 2003, PERV.

[24]  Niels Provos,et al.  ScanSSH: Scanning the Internet for SSH Servers , 2001, LISA.

[25]  Angelos D. Keromytis,et al.  The effect of DNS delays on worm propagation in an IPv6 Internet , 2005, Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies..

[26]  Christos Gkantsidis,et al.  Sampling Strategies for Epidemic-Style Information Dissemination , 2008, IEEE/ACM Transactions on Networking.

[27]  Stuart Staniford-Chen,et al.  Practical Automated Detection of Stealthy Portscans , 2002, J. Comput. Secur..

[28]  George Kurtz,et al.  Hacking Exposed: Network Security Secrets & Solutions , 1999 .

[29]  Franck Veysset,et al.  New Tool And Technique For Remote Operating System Fingerprinting , 2002 .

[30]  Kang Li,et al.  New Methods for Passive Estimation of TCP Round-Trip Times , 2005, PAM.

[31]  Sanjay Ranka,et al.  Exact Modeling of Propagation for Permutation-Scanning Worms , 2008, IEEE INFOCOM 2008 - The 27th Conference on Computer Communications.

[32]  L. Trajkovic,et al.  Mapping the Internet , 2001, Proceedings Joint 9th IFSA World Congress and 20th NAFIPS International Conference (Cat. No. 01TH8569).

[33]  Mitesh Patel,et al.  Accessing the deep web , 2007, CACM.

[34]  Stephan Mertens,et al.  Random numbers for large scale distributed Monte Carlo simulations , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[35]  Dejan S. Milojicic,et al.  SLA Decomposition: Translating Service Level Objectives to System Level Thresholds , 2007, Fourth International Conference on Autonomic Computing (ICAC'07).

[36]  Bill Cheswick,et al.  Mapping and Visualizing the Internet , 2000, USENIX Annual Technical Conference, General Track.

[37]  David Thomas,et al.  The Art in Computer Programming , 2001 .

[38]  André Trudel,et al.  World's first web census , 2007, Int. J. Web Inf. Syst..

[39]  Andreas Terzis,et al.  Fast and Evasive Attacks: Highlighting the Challenges Ahead , 2006, RAID.

[40]  David L. Black,et al.  The Addition of Explicit Congestion Notification (ECN) to IP , 2001, RFC.

[41]  Vern Paxson,et al.  How to Own the Internet in Your Spare Time , 2002, USENIX Security Symposium.

[42]  Sally Floyd,et al.  On inferring TCP behavior , 2001, SIGCOMM.

[43]  Barry Irwin,et al.  Using InetVis to Evaluate Snort and Bro Scan Detection on a Network Telescope , 2007, VizSEC.

[44]  Dawn Xiaodong Song,et al.  Distributed Evasive Scan Techniques and Countermeasures , 2007, DIMVA.

[45]  A. Langley.  Probing the viability of TCP extensions , 2008 .

[46]  Sy-Yen Kuo,et al.  Xprobe2++: Low volume remote network information gathering tool , 2009, 2009 IEEE/IFIP International Conference on Dependable Systems & Networks.

[47]  Mary K. Vernon,et al.  Mapping Internet Sensors with Probe Response Attacks , 2005, USENIX Security Symposium.