Declarative automated cloud resource orchestration

As cloud computing becomes widely deployed, one of the challenges faced involves the ability to orchestrate a highly complex set of subsystems (compute, storage, network resources) that span large geographic areas serving diverse clients. To ease this process, we present COPE (Cloud Orchestration Policy Engine), a distributed platform that allows cloud providers to perform declarative automated cloud resource orchestration. In COPE, cloud providers specify system-wide constraints and goals using COPElog, a declarative policy language geared towards specifying distributed constraint optimizations. COPE takes policy specifications and cloud system states as input and then optimizes compute, storage and network resource allocations within the cloud such that provider operational objectives and customer SLAs can be better met. We describe our proposed integration with a cloud orchestration platform, and present initial evaluation results that demonstrate the viability of COPE using production traces from a large hosting company in the US. We further discuss an orchestration scenario that involves geographically distributed data centers, and conclude with an ongoing status of our work.

[1]  Xiaozhou Li,et al.  PUMA: Policy-based Unified Multi-radio Architecture for agile mesh networking , 2012, 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012).

[2]  Boon Thau Loo,et al.  Declarative Toolkit for Rapid Network Protocol Simulation and Experimentation , 2009 .

[3]  Albert G. Greenberg,et al.  The cost of a cloud: research problems in data center networks , 2008, CCRV.

[4]  Limin Jia,et al.  Maintaining distributed logic programs incrementally , 2011, Comput. Lang. Syst. Struct..

[5]  Lili Qiu,et al.  Impact of Interference on Multi-Hop Wireless Network Performance , 2003, MobiCom '03.

[6]  Prithwish Basu,et al.  Cologne: A Declarative Distributed Constraint Optimization Platform , 2012, Proc. VLDB Endow..

[7]  Pramod Bhatotia,et al.  Orchestrating the Deployment of Computations in the Cloud with Conductor , 2012, NSDI.

[8]  Benjamin Reed,et al.  A simple totally ordered broadcast protocol , 2008, LADIS '08.

[9]  Xiaozhou Li,et al.  Declarative Policy-Based Adaptive Mobile Ad Hoc Networking , 2012, IEEE/ACM Transactions on Networking.

[10]  Xiaozhou Li,et al.  Efficient querying and maintenance of network provenance at internet-scale , 2010, SIGMOD Conference.

[11]  P. Bahl,et al.  DSAP: a protocol for coordinated spectrum access , 2005, First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005..

[12]  Ion Stoica,et al.  Declarative networking , 2009, Commun. ACM.

[13]  Philip Levis,et al.  The design and implementation of a declarative sensor network system , 2007, SenSys '07.

[14]  Yi Zhu,et al.  Heuristic methods for graph coloring problems , 2005, SAC '05.

[15]  Prashant J. Shenoy,et al.  The Case for Enterprise-Ready Virtual Private Clouds , 2009, HotCloud.

[16]  Carlo Curino,et al.  Workload-aware database monitoring and consolidation , 2011, SIGMOD '11.

[17]  Byung-Gon Chun,et al.  CloneCloud: elastic execution between mobile device and cloud , 2011, EuroSys '11.

[18]  Sanjit A. Seshia,et al.  Sketching stencils , 2007, PLDI '07.

[19]  Jacobus E. van der Merwe,et al.  Cloud Resource Orchestration: A Data-Centric Approach , 2011, CIDR.

[20]  Jeffrey D. Ullman,et al.  A Survey of Research in Deductive Database Systems , 1995 .

[21]  Eric A. Brewer,et al.  Stasis: flexible transactional storage , 2006, OSDI '06.

[22]  K. K. Ramakrishnan,et al.  Towards a ubiquitous cloud computing infrastructure , 2010, 2010 17th IEEE Workshop on Local & Metropolitan Area Networks (LANMAN).

[23]  Deep Medhi,et al.  Server Operational Cost Optimization for Cloud Computing Service Providers over a Time Horizon , 2011, Hot-ICE.

[24]  Jitendra Padhye,et al.  Routing in multi-radio, multi-hop wireless mesh networks , 2004, MobiCom '04.

[25]  Boon Thau Loo,et al.  Recursive Computation of Regions and Connectivity in Networks , 2009, 2009 IEEE 25th International Conference on Data Engineering.

[26]  Mahadev Konar,et al.  ZooKeeper: Wait-free Coordination for Internet-scale Systems , 2010, USENIX ATC.

[27]  Ion Stoica,et al.  Implementing declarative overlays , 2005, SOSP '05.

[28]  Alec Wolman,et al.  Volley: Automated Data Placement for Geo-Distributed Cloud Services , 2010, NSDI.

[29]  Xu Chen,et al.  Declarative configuration management for complex and dynamic networks , 2010, CoNEXT.

[30]  Ion Stoica,et al.  Declarative networking: language, execution and optimization , 2006, SIGMOD Conference.

[31]  Atul Singh,et al.  BFT Protocols Under Fire , 2008, NSDI.

[32]  Michael Isard,et al.  Autopilot: automatic data center management , 2007, OPSR.

[33]  Leslie Lamport,et al.  The part-time parliament , 1998, TOCS.

[34]  Jeffrey O. Kephart,et al.  An architectural approach to autonomic computing , 2004, International Conference on Autonomic Computing, 2004. Proceedings..

[35]  Jerome H. Saltzer,et al.  End-to-end arguments in system design , 1984, TOCS.

[36]  Erez Zadok,et al.  Enabling Transactional File Access via Lightweight Kernel Extensions , 2009, FAST.

[37]  Kang G. Shin,et al.  LiteGreen: Saving Energy in Networked Desktops Using Virtualization , 2010, USENIX Annual Technical Conference.

[38]  Gerald J. Popek,et al.  Transactions and Synchronization in a Distributed Operating System , 1985, SOSP.

[39]  Jonathan M. Smith,et al.  MOSAIC: Unified Platform for Dynamic Overlay Selection and Composition , 2008 .

[40]  Raghu Ramakrishnan,et al.  Database Management Systems , 1976 .

[41]  Leslie Lamport,et al.  The Byzantine Generals Problem , 1982, TOPL.

[42]  Insup Lee,et al.  DMaC: Distributed Monitoring and Checking , 2009, RV.

[43]  Tejaswi Redkar,et al.  Windows Azure Platform , 2010 .

[44]  Prithwish Basu,et al.  A declarative perspective on adaptive manet routing , 2008, PRESTO '08.

[45]  Xu Chen,et al.  ShadowNet: A Platform for Rapid and Safe Network Evolution , 2009, USENIX Annual Technical Conference.

[46]  Xiaozhou Li,et al.  Declarative policy-based adaptive MANET routing , 2009, 2009 17th IEEE International Conference on Network Protocols.

[47]  Laura Vasiliu,et al.  CloneCloud: Elastic Execution between Mobile Device and Cloud , 2012 .

[48]  Rick Bradshaw,et al.  Experiences with Eucalyptus: Deploying an Open Source Cloud , 2010, LISA.

[49]  Arun Venkataramani,et al.  Black-box and Gray-box Strategies for Virtual Machine Migration , 2007, NSDI.

[50]  Oriana Riva,et al.  Policy expressivity in the Anzere personal cloud , 2011, SoCC.

[51]  Agostino Dovier,et al.  Compiling and Executing Declarative Modeling Languages to Gecode , 2008, ICLP.

[52]  Mung Chiang,et al.  Wireless Scheduling Algorithms with O(1) Overhead for M-Hop Interference Model , 2008, 2008 IEEE International Conference on Communications.

[53]  Akhil Sahai,et al.  Automated policy-based resource construction in utility computing environments , 2004, 2004 IEEE/IFIP Network Operations and Management Symposium (IEEE Cat. No.04CH37507).

[54]  Justin Cappos,et al.  Rhizoma: A Runtime for Self-deploying, Self-managing Overlays , 2009, Middleware.

[55]  Joseph M. Hellerstein,et al.  Boom analytics: exploring data-centric, declarative programming for the cloud , 2010, EuroSys '10.

[56]  Dejan S. Milojicic,et al.  Open Cirrus TM cloud computing testbed: federated data centers for open source systems and services research , 2009, CloudCom 2009.

[57]  Donald E. Porter,et al.  Operating System Transactions , 2009, SOSP '09.

[58]  Xu Chen,et al.  TROPIC: Transactional Resource Orchestration Platform in the Cloud , 2012, USENIX Annual Technical Conference.

[59]  Prithwish Basu,et al.  Formally Verifiable Networking , 2009, HotNets.

[60]  Himanshu Gupta,et al.  Minimum Interference Channel Assignment in Multi-Radio Wireless Mesh Networks , 2007, 2007 4th Annual IEEE Communications Society Conference on Sensor, Mesh and Ad Hoc Communications and Networks.

[61]  Tzi-cker Chiueh,et al.  Centralized channel assignment and routing algorithms for multi-channel wireless mesh networks , 2004, MOCO.

[62]  Albert G. Greenberg,et al.  Optimizing Cost and Performance in Online Service Provider Networks , 2010, NSDI.

[63]  Sumit Gulwani,et al.  Spreadsheet data manipulation using examples , 2012, CACM.

[64]  David A. Maltz,et al.  Cloudward bound: planning for beneficial migration of enterprise applications to the cloud , 2010, SIGCOMM '10.

[65]  Chip Elliott,et al.  GENI - global environment for network innovations , 2008, LCN.

[66]  Thomas A. Henzinger,et al.  A marketplace for cloud resources , 2010, EMSOFT '10.

[67]  Eddie Kohler,et al.  The Click modular router , 1999, SOSP.

[68]  Lakshminarayanan Subramanian,et al.  Practical, distributed channel assignment and routing in dual-radio mesh networks , 2009, SIGCOMM '09.

[69]  Xiaozhou Li,et al.  A policy-based constraint-solving platform towards extensible wireless channel selection and routing , 2010, PRESTO '10.

[70]  Amin Vahdat,et al.  Hedera: Dynamic Flow Scheduling for Data Center Networks , 2010, NSDI.

[71]  Avi Mendelson,et al.  Programming model for a heterogeneous x86 platform , 2009, PLDI '09.

[72]  Ramana Rao Kompella,et al.  Opportunistic flooding to improve TCP transmit performance in virtualized clouds , 2011, SOCC '11.

[73]  Sharad Malik,et al.  Declarative Infrastructure Configuration Synthesis and Debugging , 2008, Journal of Network and Systems Management.

[74]  Thomas A. Henzinger,et al.  FlexPRICE: Flexible Provisioning of Resources in a Cloud Environment , 2010, 2010 IEEE 3rd International Conference on Cloud Computing.

[75]  Timothy Roscoe,et al.  Dexferizer: A service for data transfer optimization , 2011, 2011 IEEE Nineteenth IEEE International Workshop on Quality of Service.

[76]  Jeffrey O. Kephart,et al.  Autonomic computing: Architectural approach and prototype , 2006, Integr. Comput. Aided Eng..

[77]  Ion Stoica,et al.  Declarative routing: extensible routing with declarative queries , 2005, SIGCOMM '05.