The nature of data center traffic: measurements & analysis

We explore the nature of traffic in data centers, designed to support the mining of massive data sets. We instrument the servers to collect socket-level logs, with negligible performance impact. In a 1500 server operational cluster, we thus amass roughly a petabyte of measurements over two months, from which we obtain and report detailed views of traffic and congestion conditions and patterns. We further consider whether traffic matrices in the cluster might be obtained instead via tomographic inference from coarser-grained counter data.

[1]  N. McKeown,et al.  Designing a Predictable Internet Backbone Network , 2004 .

[2]  Srikanth Kandula,et al.  Dynamic load balancing without packet reordering , 2007, CCRV.

[3]  Yuan Yu,et al.  Dryad: distributed data-parallel programs from sequential building blocks , 2007, EuroSys '07.

[4]  Konstantina Papagiannaki,et al.  Traffic matrices: balancing measurements, inference and modeling , 2005, SIGMETRICS '05.

[5]  R GangerGregory,et al.  Safe and effective fine-grained TCP retransmissions for datacenter communication , 2009 .

[6]  Albert G. Greenberg,et al.  Fast accurate computation of large-scale IP traffic matrices from link loads , 2003, SIGMETRICS '03.

[7]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[8]  Ming Zhang,et al.  Understanding data center traffic characteristics , 2010, CCRV.

[9]  Rui Zhang-Shen,et al.  Designing a predictable backbone network using valiant load-balancing , 2007 .

[10]  GhemawatSanjay,et al.  The Google file system , 2003 .

[11]  Luiz André Barroso,et al.  The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines , 2009, The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines.

[12]  Albert G. Greenberg,et al.  Network anomography , 2005, IMC '05.

[13]  Mikael Johansson,et al.  Traffic matrix estimation on a large IP backbone: a comparison on real data , 2004, IMC '04.

[14]  Junda Liu,et al.  Multi-enterprise networking , 2000 .

[15]  Ramana Rao Kompella,et al.  The power of slicing in internet flow measurement , 2005, IMC '05.

[16]  Sudipta Sengupta,et al.  Efficient and robust routing of highly variable traffic , 2005 .

[17]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[18]  Carsten Lund,et al.  Estimating flow distributions from sampled flow statistics , 2005, TNET.

[19]  Saikat Guha,et al.  How healthy are today's enterprise networks? , 2008, IMC '08.

[20]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[21]  Amar Phanishayee,et al.  Safe and effective fine-grained TCP retransmissions for datacenter communication , 2009, SIGCOMM '09.

[22]  Antony I. T. Rowstron,et al.  Network exception handlers: host-network control in enterprise networks , 2008, SIGCOMM '08.

[23]  Theodore Johnson,et al.  Gigascope: a stream database for network applications , 2003, SIGMOD '03.

[24]  Jason Lee,et al.  A first look at modern enterprise traffic , 2005, IMC '05.

[25]  Jingren Zhou,et al.  SCOPE: easy and efficient parallel processing of massive data sets , 2008, Proc. VLDB Endow..

[26]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[27]  Ling Huang,et al.  Communication-Efficient Online Detection of Network-Wide Anomalies , 2007, IEEE INFOCOM 2007 - 26th IEEE International Conference on Computer Communications.

[28]  George Varghese,et al.  Building a better NetFlow , 2004, SIGCOMM.