Dynamic execution platforms over federated clouds

The increasing needs for computing power have led to parallel and distributed computing, which harness the power of large computing infrastructures in a concurrent manner. Recently, virtualization technologies have increased in popularity, thanks to hypervisors improvements, the shift to multi-core architectures, and the spread of Internet services. This has led to the emergence of cloud computing, a paradigm offering computing resources in an elastic, on-demand approach while charging only for consumed resources. In this context, this thesis proposes four contributions to leverage the power of multiple clouds. They follow two directions: the creation of elastic execution platforms on top of federated clouds, and inter-cloud live migration for using them in a dynamic manner. We propose mechanisms to efficiently build elastic execution platforms on top of multiple clouds using the sky computing federation approach. Resilin is a system for creating and managing MapReduce execution platforms on top of federated clouds, allowing to easily execute MapReduce computations without interacting with low level cloud interfaces. We propose mechanisms to reconfigure virtual network infrastructures in the presence of inter-cloud live migration, implemented in the ViNe virtual network from University of Florida. Finally, Shrinker is a live migration protocol improving the migration of virtual clusters over wide area networks by eliminating duplicated data between virtual machines.

[1]  Jie Ma,et al.  Exploiting Data Deduplication to Accelerate Live Virtual Machine Migration , 2010, 2010 IEEE International Conference on Cluster Computing.

[2]  Franck Cappello,et al.  Grid'5000: a large scale and highly reconfigurable grid experimental testbed , 2005, The 6th IEEE/ACM International Workshop on Grid Computing, 2005..

[3]  David Abramson,et al.  High performance parametric modeling with Nimrod/G: killer application for the global grid? , 2000, Proceedings 14th International Parallel and Distributed Processing Symposium. IPDPS 2000.

[4]  M. Radenkovic Usre Proxy Service in Mygrid. , 2003 .

[5]  Yingwei Luo,et al.  Live and incremental whole-system migration of virtual machines using block-bitmap , 2008, 2008 IEEE International Conference on Cluster Computing.

[6]  Franck Cappello,et al.  MPI versus MPI+OpenMP on the IBM SP for the NAS Benchmarks , 2000, ACM/IEEE SC 2000 Conference (SC'00).

[7]  Eugene Ciurana,et al.  Google App Engine , 2009 .

[8]  Anja Feldmann,et al.  Live wide-area migration of virtual machines including local persistent state , 2007, VEE '07.

[9]  Scott Shenker,et al.  Internet indirection infrastructure , 2004, IEEE/ACM Transactions on Networking.

[10]  Tejaswi Redkar,et al.  Windows Azure Platform , 2010 .

[11]  Hai Jin,et al.  Live virtual machine migration with adaptive, memory compression , 2009, 2009 IEEE International Conference on Cluster Computing and Workshops.

[12]  Pierre Riteau Building Dynamic Computing Infrastructures over Distributed Clouds , 2011, NCCA.

[13]  James E. Smith,et al.  The architecture of virtual machines , 2005, Computer.

[14]  Andrew S. Grimshaw,et al.  Legion: Lessons Learned Building a Grid Operating System , 2005, Proceedings of the IEEE.

[15]  Sean Quinlan,et al.  Venti: A New Approach to Archival Storage , 2002, FAST.

[16]  Judy Qiu,et al.  A hierarchical framework for cross-domain MapReduce execution , 2011, ECMLS '11.

[17]  Steven Hand,et al.  Satori: Enlightened Page Sharing , 2009, USENIX Annual Technical Conference.

[18]  Marvin Theimer,et al.  Preemptable remote execution facilities for the V-system , 1985, SOSP '85.

[19]  Hai Jin,et al.  Live migration of virtual machine based on full system trace and replay , 2009, HPDC '09.

[20]  Rusty Russell,et al.  virtio: towards a de-facto standard for virtual I/O devices , 2008, OPSR.

[21]  Jack Dongarra,et al.  MPI: The Complete Reference , 1996 .

[22]  Gerald J. Popek,et al.  Formal requirements for virtualizable third generation architectures , 1974, SOSP '73.

[23]  Renato Figueiredo,et al.  Science Clouds: Early Experiences in Cloud Computing for Scientific Applications , 2008 .

[24]  Dejan S. Milojicic,et al.  Process migration , 1999, ACM Comput. Surv..

[25]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[26]  Lavanya Ramakrishnan,et al.  Magellan: experiences from a science cloud , 2011, ScienceCloud '11.

[27]  Tal Garfinkel,et al.  The Design and Evolution of Live Storage Migration in VMware ESX , 2011, USENIX Annual Technical Conference.

[28]  Christine Morin,et al.  Shrinker: Efficient Wide-Area Live Virtual Machine Migration using Distributed Content-Based Addressing , 2010 .

[29]  Giorgio Ventre,et al.  System-Level Virtualization and Mobile IP to Support Service Mobility , 2009, 2009 International Conference on Parallel Processing Workshops.

[30]  Xiaoyun Wang,et al.  Finding Collisions in the Full SHA-1 , 2005, CRYPTO.

[31]  Sebastien Goasguen,et al.  Image Distribution Mechanisms in Large Scale Cloud Providers , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[32]  Eyal de Lara,et al.  SnowFlock: rapid virtual machine cloning for cloud computing , 2009, EuroSys '09.

[33]  Gabriel Antoniu,et al.  BlobSeer: Next-generation data management for large scale infrastructures , 2011, J. Parallel Distributed Comput..

[34]  Fang Hao,et al.  Enhancing dynamic cloud-based services using network virtualization , 2009, CCRV.

[35]  Rajkumar Buyya,et al.  InterCloud: Utility-Oriented Federation of Cloud Computing Environments for Scaling of Application Services , 2010, ICA3PP.

[36]  Daniel Price,et al.  Solaris Zones: Operating System Support for Consolidating Commercial Workloads , 2004, LISA.

[37]  Christine Morin,et al.  Towards XtreemOS in the Clouds – Automatic Deployment of XtreemOS Resources in a Nimbus Cloud , 2010 .

[38]  Huan Liu,et al.  Cutting MapReduce Cost with Spot Market , 2011, HotCloud.

[39]  José A. B. Fortes,et al.  User-Level Virtual Network Support for Sky Computing , 2009, 2009 Fifth IEEE International Conference on e-Science.

[40]  Richard Wolski,et al.  The Eucalyptus Open-Source Cloud-Computing System , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[41]  Petter Svärd,et al.  Evaluation of delta compression techniques for efficient live migration of large virtual machines , 2011, VEE '11.

[42]  Henri Casanova,et al.  SimGrid: A Generic Framework for Large-Scale Distributed Experiments , 2008, Tenth International Conference on Computer Modeling and Simulation (uksim 2008).

[43]  Randy H. Katz,et al.  Above the Clouds: A Berkeley View of Cloud Computing , 2009 .

[44]  Charles E. Perkins,et al.  Mobility support in IPv6 , 1996, MobiCom '96.

[45]  John Shalf,et al.  Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[46]  José A. B. Fortes,et al.  Sky Computing on FutureGrid and Grid'5000 , 2010 .

[47]  Robert P. Goldberg,et al.  Survey of virtual machine research , 1974, Computer.

[48]  Peter Desnoyers,et al.  Memory buddies: exploiting page sharing for smart colocation in virtualized data centers , 2009, VEE '09.

[49]  Michael C. Schatz,et al.  CloudBurst: highly sensitive read mapping with MapReduce , 2009, Bioinform..

[50]  Anthony Liguori,et al.  Experiences with Content Addressable Storage and Virtual Disks , 2008, Workshop on I/O Virtualization.

[51]  Satoshi Sekiguchi,et al.  A live storage migration mechanism over wan and its performance evaluation , 2009, VTDC '09.

[52]  Thilo Kielmann,et al.  Budget Estimation and Control for Bag-of-Tasks Scheduling in Clouds , 2011, Parallel Process. Lett..

[53]  Renato J. O. Figueiredo,et al.  A case for grid computing on virtual machines , 2003, 23rd International Conference on Distributed Computing Systems, 2003. Proceedings..

[54]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[55]  Chenyu Wang,et al.  Exploring MapReduce efficiency with highly-distributed data , 2011, MapReduce '11.

[56]  Mike Murphy,et al.  The Efficacy of Live Virtual Machine Migrations Over the Internet , 2007, Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing (VTDC '07).

[57]  Larry Rudolph,et al.  Towards Convergence in Job Schedulers for Parallel Supercomputers , 1996, JSSPP.

[58]  Christine Morin,et al.  Bringing Elastic MapReduce to Scientific Clouds , 2011, CloudCom 2011.

[59]  Rubén S. Montero,et al.  Multicloud Deployment of Computing Clusters for Loosely Coupled MTC Applications , 2011, IEEE Transactions on Parallel and Distributed Systems.

[60]  Umesh Deshpande,et al.  Live gang migration of virtual machines , 2011, HPDC '11.

[61]  Djawida Dib Migration dynamique d'applications réparties virtualisées dans les fédérations d'infrastructures distribuées , 2010 .

[62]  Shishir Garg,et al.  Opening the clouds: qualitative overview of the state-of-the-art open source VM-based cloud management platforms , 2009, Middleware.

[63]  Gil Neiger,et al.  Intel virtualization technology , 2005, Computer.

[64]  George Varghese,et al.  Difference engine , 2010, OSDI.

[65]  Mahadev Satyanarayanan,et al.  Internet suspend/resume , 2002, Proceedings Fourth IEEE Workshop on Mobile Computing Systems and Applications.

[66]  Katarzyna Keahey,et al.  Contextualization: Providing One-Click Virtual Clusters , 2008, 2008 IEEE Fourth International Conference on eScience.

[67]  P. Oscar Boykin,et al.  IP over P2P: enabling self-configuring virtual IP networks for grid computing , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[68]  Joe Weinman,et al.  The future of Cloud Computing , 2011, 2011 IEEE Technology Time Machine Symposium on Technologies Beyond 2020.

[69]  David F. Heidel,et al.  An Overview of the BlueGene/L Supercomputer , 2002, ACM/IEEE SC 2002 Conference (SC'02).

[70]  José A. B. Fortes,et al.  Crosscloud Computing , 2010, High Performance Computing Workshop.

[71]  Klaus Wehrle,et al.  OCALA: An Architecture for Supporting Legacy Applications over Overlays , 2006, NSDI.

[72]  Charles E. Perkins,et al.  IP Mobility Support for IPv4, Revised , 2010, RFC.

[73]  Paul Hudak,et al.  Memory coherence in shared virtual memory systems , 1989, TOCS.

[74]  Dutch T. Meyer,et al.  Remus: High Availability via Asynchronous Virtual Machine Replication. (Best Paper) , 2008, NSDI.

[75]  John Bresnahan,et al.  Managing appliance launches in infrastructure clouds , 2011, TG.

[76]  John Bresnahan,et al.  Cumulus: an open source storage cloud for science , 2011, ScienceCloud '11.

[77]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[78]  Hairong Kuang,et al.  The Hadoop Distributed File System , 2010, 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).

[79]  Dean H. Lorenz,et al.  IP mobility to support live migration of virtual machines across subnets , 2009, SYSTOR '09.

[80]  Chandra Krintz,et al.  Evaluating the Performance Impact of Xen on MPI and Process Execution For HPC Systems , 2006, First International Workshop on Virtualization Technology in Distributed Computing (VTDC 2006).

[81]  Christine Morin,et al.  Clouds: a New Playground for the XtreemOS Grid Operating System , 2009, Parallel Process. Lett..

[82]  David P. Anderson,et al.  BOINC: a system for public-resource computing and storage , 2004, Fifth IEEE/ACM International Workshop on Grid Computing.

[83]  Bofeng Zhang,et al.  Comparison of Several Cloud Computing Platforms , 2009, 2009 Second International Symposium on Information Science and Engineering.

[84]  Christian Engelmann,et al.  Proactive fault tolerance for HPC with Xen virtualization , 2007, ICS '07.

[85]  Ole Agesen,et al.  A comparison of software and hardware techniques for x86 virtualization , 2006, ASPLOS XII.

[86]  Olivier Richard,et al.  TakTuk, adaptive deployment of remote executions , 2009, HPDC '09.

[87]  Christine Morin,et al.  Shrinker: Improving Live Migration of Virtual Clusters over WANs with Distributed Data Deduplication and Content-Based Addressing , 2011, Euro-Par.

[88]  Fabienne Anhalt,et al.  Linux-based virtualization for HPC clusters , 2009 .

[89]  Andrew Birrell,et al.  Implementing remote procedure calls , 1984, TOCS.

[90]  Andrew Warfield,et al.  Live migration of virtual machines , 2005, NSDI.

[91]  Dejan S. Milojicic,et al.  Open Cirrus TM cloud computing testbed: federated data centers for open source systems and services research , 2009, CloudCom 2009.

[92]  Benoit Hudzia,et al.  Improving the live migration process of large enterprise applications , 2009, VTDC '09.

[93]  José A. B. Fortes,et al.  Sky Computing , 2009, IEEE Internet Computing.

[94]  Hidenobu Watanabe,et al.  A Performance Improvement Method for the Global Live Migration of Virtual Machine with IP Mobility , 2010 .

[95]  Himabindu Pucha,et al.  Towards Optimizing Hadoop Provisioning in the Cloud , 2009, HotCloud.

[96]  Antonio Puliafito,et al.  How to Enhance Cloud Architectures to Enable Cross-Federation , 2010, IEEE CLOUD.

[97]  Christine Morin,et al.  Towards an efficient single system image cluster operating system , 2002 .

[98]  Huan Liu,et al.  Cloud MapReduce: A MapReduce Implementation on Top of a Cloud Operating System , 2011, 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.

[99]  Ian T. Foster,et al.  Virtual workspaces: Achieving quality of service and quality of life in the Grid , 2005, Sci. Program..

[100]  D. Ogrizovic,et al.  Open source science clouds , 2010, The 33rd International Convention MIPRO.

[101]  Aleksey Pesterev,et al.  Fast, Inexpensive Content-Addressed Storage in Foundation , 2008, USENIX Annual Technical Conference.

[102]  Andy Hopper,et al.  Predicting the Performance of Virtual Machine Migration , 2010, 2010 IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems.

[103]  Xiaomin Zhu,et al.  From virtualized resources to virtual computing grids: the In-VIGO system , 2005, Future Gener. Comput. Syst..

[104]  Geoffrey C. Fox,et al.  MapReduce in the Clouds for Science , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[105]  Shujia Zhou,et al.  Case study for running HPC applications in public clouds , 2010, HPDC '10.

[106]  Satoshi Sekiguchi,et al.  Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension , 2010, 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing.

[107]  Stephen A. Rago,et al.  Advanced Programming in the UNIX(R) Environment (2nd Edition) , 2005 .

[108]  Benny Rochwerger,et al.  Inter-cloud mobility of virtual machines , 2011, SYSTOR '11.

[109]  Mahadev Satyanarayanan,et al.  Design Tradeoffs in Applying Content Addressable Storage to Enterprise-scale Systems Based on Virtual Machines , 2006, USENIX Annual Technical Conference, General Track.

[110]  Xuxian Jiang,et al.  VIOLIN: Virtual Internetworking on Overlay Infrastructure , 2004, ISPA.

[111]  José A. B. Fortes,et al.  CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications , 2008, 2008 IEEE Fourth International Conference on eScience.

[112]  E. Myers,et al.  Basic local alignment search tool. , 1990, Journal of molecular biology.

[113]  Monica S. Lam,et al.  Optimizing the migration of virtual computers , 2002, OPSR.

[114]  C. Waldspurger Memory resource management in VMware ESX server , 2002, OSDI '02.

[115]  Thomas C. Bressoud,et al.  Using Content Addressing to Transfer Virtual Machine State , 2002 .

[116]  Eugenio Cesario,et al.  The XtreemFS architecture—a case for object‐based file systems in Grids , 2008, Concurr. Comput. Pract. Exp..

[117]  Asser N. Tantawi,et al.  See Spot Run: Using Spot Instances for MapReduce Workflows , 2010, HotCloud.

[118]  Douglas Thain,et al.  A Comparison and Critique of Eucalyptus, OpenNebula and Nimbus , 2010, 2010 IEEE Second International Conference on Cloud Computing Technology and Science.

[119]  Satoshi Sekiguchi,et al.  A Live Storage Migration Mechanism over WAN for Relocatable Virtual Machine Services on Clouds , 2009, 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid.

[120]  Xavier Lorca,et al.  Entropy: a consolidation manager for clusters , 2009, VEE '09.

[121]  Chris Rose,et al.  A Break in the Clouds: Towards a Cloud Definition , 2011 .

[122]  Gabriel Antoniu,et al.  Going back and forth: efficient multideployment and multisnapshotting on clouds , 2011, HPDC '11.

[123]  Christine Morin,et al.  OpenMosix, OpenSSI and Kerrighed: a comparative study , 2005, CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005..

[124]  Kartik Gopalan,et al.  Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning , 2009, VEE '09.

[125]  Christine Morin,et al.  Virtual Organization Support within a Grid-Wide Operating System , 2008, IEEE Internet Computing.

[126]  Konstantin V. Shvachko Apache Hadoop: The Scalability Update , 2011, login Usenix Mag..

[127]  José A. B. Fortes,et al.  Large-Scale Cloud Computing Research: Sky Computing on FutureGrid and Grid'5000 , 2010, ERCIM News.

[128]  Fabrice Bellard,et al.  QEMU, a Fast and Portable Dynamic Translator , 2005, USENIX ATC, FREENIX Track.

[129]  Ian Foster,et al.  The Grid 2 - Blueprint for a New Computing Infrastructure, Second Edition , 1998, The Grid 2, 2nd Edition.

[130]  Zheng Shao,et al.  Data warehousing and analytics infrastructure at facebook , 2010, SIGMOD Conference.

[131]  Roy T. Fielding,et al.  Principled design of the modern Web architecture , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[132]  Paul Marshall,et al.  Elastic Site: Using Clouds to Elastically Extend Site Resources , 2010, 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing.

[133]  Larry L. Peterson,et al.  Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors , 2007, EuroSys '07.

[134]  Muli Ben-Yehuda,et al.  The Reservoir model and architecture for open federated cloud computing , 2009, IBM J. Res. Dev..

[135]  Christine Morin,et al.  Resilin: Elastic MapReduce for Private and Community Clouds , 2011 .

[136]  Eyal de Lara,et al.  SnowFlock: Virtual Machine Cloning as a First-Class Cloud Primitive , 2011, TOCS.

[137]  GhemawatSanjay,et al.  The Google file system , 2003 .

[138]  Craig A. Knoblock,et al.  Advanced Programming in the UNIX Environment , 1992, Addison-Wesley professional computing series.

[139]  Pengcheng Liu,et al.  Heterogeneous Live Migration of Virtual Machines , 2008 .

[140]  Rajkumar Buyya,et al.  Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters , 2009, HPDC '09.

[141]  A. Kivity,et al.  kvm : the Linux Virtual Machine Monitor , 2007 .

[142]  Ewa Deelman,et al.  Experiences using cloud computing for a scientific workflow application , 2011, ScienceCloud '11.

[143]  David E. Culler,et al.  A blueprint for introducing disruptive technology into the Internet , 2003, CCRV.

[144]  Bernd Freisleben,et al.  Efficient Distribution of Virtual Machines for Cloud Computing , 2010, 2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing.

[145]  Xianghua Xu,et al.  Quantifying Performance Properties of Virtual Machine , 2008, 2008 International Symposium on Information Science and Engineering.

[146]  Pierre Riteau,et al.  User-level virtual networking mechanisms to support virtual machine migration over multiple clouds , 2010, 2010 IEEE Globecom Workshops.

[147]  Leon Gommans,et al.  Seamless live migration of virtual machines over the MAN/WAN , 2006, Future Gener. Comput. Syst..

[148]  Charles E. Perkins,et al.  IP Mobility Support for IPv4 , 2002, RFC.

[149]  José A. B. Fortes,et al.  A virtual network (ViNe) architecture for grid computing , 2006, Proceedings 20th IEEE International Parallel & Distributed Processing Symposium.

[150]  Edward Walker,et al.  Benchmarking Amazon EC2 for High-Performance Scientific Computing , 2008, login Usenix Mag..

[151]  Amnon Barak,et al.  The MOSIX multicomputer operating system for high performance cluster computing , 1998, Future Gener. Comput. Syst..

[152]  Borja Sotomayor,et al.  Virtual Infrastructure Management in Private and Hybrid Clouds , 2009, IEEE Internet Computing.

[153]  Christian Engelmann,et al.  Proactive Fault Tolerance Using Preemptive Migration , 2009, 2009 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing.