Choreo: network-aware task placement for cloud applications

Cloud computing infrastructures are increasingly being used by network-intensive applications that transfer significant amounts of data between the nodes on which they run. This paper shows that tenants can do a better job placing applications by understanding the underlying cloud network as well as the demands of the applications. To do so, tenants must be able to quickly and accurately measure the cloud network and profile their applications, and then use a network-aware placement method to place applications. This paper describes Choreo, a system that solves these problems. Our experiments measure Amazon's EC2 and Rackspace networks and use three weeks of network data from applications running on the HP Cloud network. We find that Choreo reduces application completion time by an average of 8%-14% (max improvement: 61%) when applications are placed all at once, and 22%-43% (max improvement: 79%) when they arrive in real-time, compared to alternative placement schemes.

[1]  Ling Liu,et al.  Purlieus: Locality-aware resource allocation for MapReduce in a cloud , 2011, 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (SC).

[2]  Hitesh Ballani,et al.  Towards predictable datacenter networks , 2011, SIGCOMM 2011.

[3]  A. Adas,et al.  Traffic models in broadband networks , 1997, IEEE Commun. Mag..

[4]  Costin Raiciu,et al.  Opening Up Black Box Networks with CloudTalk , 2012, HotCloud.

[5]  David A. Maltz,et al.  Network traffic characteristics of data centers in the wild , 2010, IMC '10.

[6]  Qi He,et al.  On the predictability of large transfer TCP throughput , 2005, SIGCOMM '05.

[7]  Rob Sherwood,et al.  Discarte: a disjunctive internet cartographer , 2008, SIGCOMM '08.

[8]  Jean-Chrysostome Bolot,et al.  Characterizing End-to-End Packet Delay and Loss in the Internet , 1993, J. High Speed Networks.

[9]  T. S. Eugene Ng,et al.  The Impact of Virtualization on Network Performance of Amazon EC2 Data Center , 2010, 2010 Proceedings IEEE INFOCOM.

[10]  Ashraf Aboulnaga,et al.  Deploying Database Appliances in the Cloud , 2009, IEEE Data Eng. Bull..

[11]  Jorge-Arnulfo Quiané-Ruiz,et al.  Runtime measurements in the cloud , 2010, Proc. VLDB Endow..

[12]  Matthew Mathis,et al.  The macroscopic behavior of the TCP congestion avoidance algorithm , 1997, CCRV.

[13]  Vern Paxson,et al.  End-to-end Internet packet dynamics , 1997, SIGCOMM '97.

[14]  Ming Zhang,et al.  Understanding data center traffic characteristics , 2010, CCRV.

[15]  Naveen Sharma,et al.  Towards autonomic workload provisioning for enterprise Grids and clouds , 2009, 2009 10th IEEE/ACM International Conference on Grid Computing.

[16]  Xiaowei Yang,et al.  CloudCmp: comparing public cloud providers , 2010, IMC '10.

[17]  Ratul Mahajan,et al.  Measuring ISP topologies with Rocketfuel , 2004, IEEE/ACM Transactions on Networking.

[18]  Raj Jain,et al.  Packet Trains-Measurements and a New Model for Computer Network Traffic , 1986, IEEE J. Sel. Areas Commun..

[19]  Manish Jain,et al.  End-to-end estimation of the available bandwidth variation range , 2005, SIGMETRICS '05.

[20]  Albert G. Greenberg,et al.  A flexible model for resource management in virtual private networks , 1999, SIGCOMM '99.

[21]  Albert G. Greenberg,et al.  VL2: a scalable and flexible data center network , 2009, SIGCOMM '09.

[22]  Manish Jain,et al.  Ten fallacies and pitfalls on end-to-end available bandwidth estimation , 2004, IMC '04.

[23]  Albert G. Greenberg,et al.  Sharing the Data Center Network , 2011, NSDI.

[24]  Amin Vahdat,et al.  Hedera: Dynamic Flow Scheduling for Data Center Networks , 2010, NSDI.

[25]  Saeed Rasouli Heikalabad,et al.  A Novel Virtual Machine Placement in Cloud Computing , 2011 .

[26]  Hari Balakrishnan,et al.  A Plan for Optimizing Network-Intensive Cloud Applications , 2013 .

[27]  Thomas E. Anderson,et al.  Reverse traceroute , 2010, NSDI.

[28]  kc claffy,et al.  Bandwidth estimation: metrics, measurement techniques, and tools , 2003, IEEE Netw..