Exploring Visible Internet Hosts through Census and Survey USC / ISI Technical Report ISI-TR-2007-640

Measurement studies published in the literature have, for the most part, ignored the population of hosts. While many hosts are hidden behind firewalls and in private address space, there is much to be learned from examining the population of visible Internet hosts—one can better understand network growth and accessibility and this understanding ca n help to assess vulnerabilities, deployment of new technolo gies, and improve network models. This paper is, to our knowledge, the first attempt to measure the population of visible Internet edge hosts. We measure hosts in two ways: via periodic Internet censuses, where we queryall accessible Internet addresses every few months, and viasurveysof a small fraction of the responsive address space, probing each address every 11 minutes for one week. These approaches are complementary: a census is effective at evaluating the Internet as a whole, while surveys validat e the census and allow observation of the lifetime of typical address occupancy. We find that only 3.6% of allocated addresses are actually occupied by visible hosts, and that occupancy is unevenly distributed, with a quarter of responsive /24 subnets less t han 5% full, and only 9% of subnets more than half full. We establish an upper-bound on the number of servers in the Internet at 36 million, about 16% of the responsive addresses. Many firewalls are visible and we observe significant diversity in the distribution of firewalled block size. While the absolute number of firewalled blocks appears stable, the ratio of coverage of visible firewalls to the number of visible addresses is declining, perhaps suggesting increasing use of invisible firewalls.

[1]  Nicola Jones.  Design and Inference in Finite Population Sampling , 1993 .

[2]  Vince Fuller,et al.  Classless Inter-Domain Routing (CIDR): an Address Assignment and Aggregation Strategy , 1993, RFC.

[3]  Tony L. Eng,et al.  Extending the IP internet through address reuse , 1993, CCRV.

[4]  Walter Willinger,et al.  On the self-similar nature of Ethernet traffic , 1993, SIGCOMM '93.

[5]  Yakov Rekhter,et al.  Address Allocation for Private Internets , 1994, RFC.

[6]  Vern Paxson,et al.  End-to-end Internet packet dynamics , 1997, SIGCOMM '97.

[7]  Vern Paxson.  End-to-end internet packet dynamics , 1999, TNET.

[8]  Michalis Faloutsos,et al.  On power-law relationships of the Internet topology , 1999, SIGCOMM '99.

[9]  Ramesh Govindan,et al.  Heuristics for Internet map discovery , 2000, Proceedings IEEE INFOCOM 2000. Conference on Computer Communications. Nineteenth Annual Joint Conference of the IEEE Computer and Communications Societies (Cat. No.00CH37064).

[10]  Farnam Jahanian,et al.  Defeating TCP/IP Stack Fingerprinting , 2000, USENIX Security Symposium.

[11]  M. Grossglauser,et al.  Trajectory sampling for direct traffic observation , 2000 .

[12]  Rob Beck.  Passive-aggressive resistance: OS fingerprint evasion , 2001 .

[13]  Abhijit Bose,et al.  Delayed Internet routing convergence , 2000, SIGCOMM.

[14]  Lixin Gao.  On inferring autonomous system relationships in the internet , 2001, TNET.

[15]  Kevin Jeffay,et al.  What TCP/IP protocol headers can tell us about the web , 2001, SIGMETRICS '01.

[16]  K. Claffy,et al.  Topology discovery by active probing , 2002, Proceedings 2002 Symposium on Applications and the Internet (SAINT) Workshops.

[17]  S. Shenker,et al.  Network topology generators: degree-based vs. structural , 2002, SIGCOMM '02.

[18]  Randy H. Katz,et al.  Characterizing the Internet hierarchy from multiple vantage points , 2002, Proceedings.Twenty-First Annual Joint Conference of the IEEE Computer and Communications Societies.

[19]  Krishna P. Gummadi,et al.  Measurement, modeling, and analysis of a peer-to-peer file-sharing workload , 2003, SOSP '03.

[20]  George Varghese,et al.  The impact of address allocation and routing on the structure and implementation of routing tables , 2003, SIGCOMM '03.

[21]  Mark Allman,et al.  Estimating loss rates with TCP , 2003, PERV.

[22]  Ratul Mahajan,et al.  Measuring ISP topologies with Rocketfuel , 2004, IEEE/ACM Transactions on Networking.

[23]  Walter Willinger,et al.  A first-principles approach to understanding the internet's router-level topology , 2004, SIGCOMM '04.

[24]  Ratul Mahajan,et al.  Measuring ISP topologies with rocketfuel , 2002, TNET.

[25]  Don Towsley,et al.  On characterizing BGP routing table growth , 2004, Comput. Networks.

[26]  Tony Hain.  A Pragmatic Report on IPv4 Address Space Consumption , 2005 .

[27]  Songwu Lu,et al.  IPv4 address allocation and the BGP routing table evolution , 2005, CCRV.

[28]  Yuval Shavitt,et al.  DIMES: let the internet measure itself , 2005, CCRV.

[29]  Anja Feldmann,et al.  Building an AS-topology model that captures route diversity , 2006, SIGCOMM 2006.

[30]  Eddie Kohler,et al.  Observed structure of addresses in IP traffic , 2006, TNET.

[31]  Dmitri V. Krioukov,et al.  AS relationships: inference and validation , 2006, CCRV.

[32]  M. Goldszmidt,et al.  How dynamic are IP addresses? , 2007, SIGCOMM '07.