Global public computing

High-bandwidth networking and cheap computing hardware are leading to a world in which the resources of one machine are available to groups of users beyond their immediate owner. This trend is visible in many different settings. Distributed computing, where applications are divided into parts that run on different machines for load distribution, geographical dispersion, or robustness, has recently found new fertile ground. Grid computing promises to provide a common framework for scheduling scientific computation and managing the associated large data sets. Proposals for utility computing envision a world in which businesses rent computing bandwidth in server farms on-demand instead of purchasing and maintaining servers themselves. All such architectures target particular user and application groups or deployment scenarios, where simplifying assumptions can be made. They expect centralised ownership of resources, cooperative users, and applications that are well-behaved and compliant to a specific API or middleware. Members of the public who are not involved in Grid communities or wish to deploy out-of-the-box distributed services, such as game servers, have no means to acquire resources on large numbers of machines around the world to launch their tasks. This dissertation proposes a new distributed computing paradigm, termed global public computing, which allows any user to run any code anywhere. Such platforms price computing resources, and ultimately charge users for resources consumed. This dissertation presents the design and implementation of the Xeno-Server Open Platform, putting this vision into practice. The efficiency and scala-bility of the developed mechanisms are demonstrated by experimental evaluation; the prototype platform allows the global-scale deployment of complex services in less than 45 seconds, and could scale to millions of concurrent sessions without presenting performance bottlenecks. To facilitate global public computing, this work addresses several research challenges. It introduces reusable mechanisms for representing, advertising, and supporting the discovery of resources. To allow flexible and federated control of resource allocation by all stakeholders involved, it proposes a novel role-based resource management framework for expressing and combining distributed management policies. Furthermore, it implements effective service deployment models for launching distributed services on large numbers of machines around the world easily, quickly, and efficiently. To keep track of resource consumption and pass charges on to consumers, it devises an accounting and charging infrastructure. 3 4 Acknowledgements Although a doctoral dissertation is always the result of largely solitary work, I have been truly fortunate to have cooperated with many outstanding people, to whom I express my …

[1]  Andrew Warfield,et al.  Live migration of virtual machines , 2005, NSDI.

[2]  Douglas Thain,et al.  Distributed computing in practice: the Condor experience , 2005, Concurr. Pract. Exp..

[3]  F. Comín The Truth About Markets. Their Genius, their Limits, their Follies, John Kay. Allen Lane, London (2003), Culture and Prosperity. The Truth About Markets, why some nations are rich but most remain poor,, New York, Harper-Collins, 2004, 420 pp.] , 2005 .

[4]  D. Schroeder The Truth about Markets. Their Genius, their Limits, their Follies by John Kay , 2005 .

[5]  Keir Fraser,et al.  Global-scale Service Deployment in the XenoServer Platform , 2004, WORLDS.

[6]  Matjaz B. Juric,et al.  Business process execution language for web services , 2004 .

[7]  Laurent Massoulié,et al.  Faithfulness in internet algorithms , 2004, PINS '04.

[8]  Stephan Reiff-Marganiec,et al.  Feature interaction in policies , 2004, Comput. Networks.

[9]  Ian Pratt,et al.  Proceedings of the General Track: 2004 USENIX Annual Technical Conference , 2004 .

[10]  Anne H. Anderson An introduction to the Web Services Policy Language (WSPL) , 2004, Proceedings. Fifth IEEE International Workshop on Policies for Distributed Systems and Networks, 2004. POLICY 2004..

[11]  Michel R. V. Chaudron,et al.  Prediction of Run-Time Resource Consumption in Multi-task Component-Based Software Systems , 2004, CBSE.

[12]  Sunil Chandra,et al.  Decentralized orchestration of composite web services , 2004, WWW Alt. '04.

[13]  Evangelos Kotsovinos,et al.  Pinocchio: Incentives for Honest Participation in Distributed Trust Management , 2004, iTrust.

[14]  David E. Culler,et al.  Operating Systems Support for Planetary-Scale Network Services , 2004, NSDI.

[15]  Scott Shenker,et al.  Spurring Adoption of DHTs with OpenHash, a Public DHT Service , 2004, IPTPS.

[16]  Cecilia Mascolo,et al.  XMILE: An XML Based Approach for Incremental Code Mobility and Update , 2002, Automated Software Engineering.

[17]  Angela Orebaugh Ethereal packet sniffing , 2004 .

[18]  Dengguo Feng,et al.  Collisions for Hash Functions MD4, MD5, HAVAL-128 and RIPEMD , 2004, IACR Cryptol. ePrint Arch..

[19]  Amin Vahdat,et al.  Distributed Resource Discovery on PlanetLab with SWORD , 2004, WORLDS.

[20]  Boris Motik,et al.  Managing multiple and distributed ontologies on the Semantic Web , 2003, The VLDB Journal.

[21]  Dennis G. Kafura,et al.  First experiences using XACML for access control in distributed systems , 2003, XMLSEC '03.

[22]  HarrisTim,et al.  Xen and the art of virtualization , 2003 .

[23]  Peter R. Pietzuch,et al.  XenoTrust: event-based distributed trust management , 2003, 14th International Workshop on Database and Expert Systems Applications, 2003. Proceedings..

[24]  David E. Culler,et al.  PlanetLab: an overlay testbed for broad-coverage services , 2003, CCRV.

[25]  Rajesh Raman,et al.  Policy driven heterogeneous resource co-allocation with Gangmatching , 2003, High Performance Distributed Computing, 2003. Proceedings. 12th IEEE International Symposium on.

[26]  Timothy L. Harris,et al.  XenoSearch: distributed resource discovery in the XenoServer open platform , 2003, High Performance Distributed Computing, 2003. Proceedings. 12th IEEE International Symposium on.

[27]  Steven Hand,et al.  Managing Trust and Reputation in the XenoServer Open Platform , 2003, iTrust.

[28]  Steven Hand,et al.  Controlling the XenoServer Open Platform , 2003, 2003 IEEE Conference onOpen Architectures and Network Programming..

[29]  Ben Y. Zhao,et al.  Pond: The OceanStore Prototype , 2003, FAST.

[30]  Oliver Heckmann,et al.  A market managed multi-service Internet (M3I) , 2003, Comput. Commun..

[31]  海洋研究開発機構横浜研究所地球シミュレータセンター,et al.  Annual report of the Earth Simulator Center , 2003 .

[32]  Von Welch,et al.  Fine-Grain Authorization Policies in the GRID: Design and Implementation , 2003, Middleware Workshops.

[33]  Rajesh Raman,et al.  Distributed Policy Management and Comprehension with Classified Advertisements , 2003 .

[34]  David E. Culler,et al.  A blueprint for introducing disruptive technology into the Internet , 2003, CCRV.

[35]  Tim Harris,et al.  Role-Based Resource Management , 2003 .

[36]  Brent N. Chun,et al.  Slice Creation and Management , 2003 .

[37]  Tony Andrews Business Process Execution Language for Web Services Version 1.1 , 2003 .

[38]  Keir Fraser,et al.  The Xenoserver computing infrastructure , 2003 .

[39]  Peter Triantafillou,et al.  Towards High Performance Peer-to-Peer Content and Resource Sharing Systems , 2003, CIDR.

[40]  Mike Hibler,et al.  USENIX Association Proceedings of the General Track : 2003 USENIX Annual , 2003 .

[41]  Marianne Shaw,et al.  Scale and performance in the Denali isolation kernel , 2002, OSDI '02.

[42]  Jeff Hodges,et al.  Lightweight Directory Access Protocol (v3): Technical Specification , 2002, RFC.

[43]  Peter R. Pietzuch,et al.  Hermes: a distributed event-based middleware architecture , 2002, Proceedings 22nd International Conference on Distributed Computing Systems Workshops.

[44]  Mahadev Satyanarayanan,et al.  Internet suspend/resume , 2002, Proceedings Fourth IEEE Workshop on Mobile Computing Systems and Applications.

[45]  Emil C. Lupu,et al.  An adaptive policy based management framework for differentiated services networks , 2002, Proceedings Third International Workshop on Policies for Distributed Systems and Networks.

[46]  Timothy L. Harris,et al.  Storage, Mutability and Naming in Pasta , 2002, NETWORKING Workshops.

[47]  Ninghui Li,et al.  Design of a role-based trust-management framework , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[48]  James Snell,et al.  Introduction to Web services architecture , 2002, IBM Syst. J..

[49]  John R. Douceur,et al.  The Sybil Attack , 2002, IPTPS.

[50]  David Wetherall,et al.  Towards an active network architecture , 1996, Proceedings DARPA Active Networks Conference and Exposition.

[51]  Mendel Rosenblum,et al.  The vMatrix: A Network of Virtual Machine Monitors for Dynamic Content Distribution , 2002 .

[52]  Steven Tuecke,et al.  The Physiology of the Grid An Open Grid Services Architecture for Distributed Systems Integration , 2002 .

[53]  Atanu Ghosh,et al.  Towards the rapid network-wide deployment of new application specific network protocols, using application level active networking , 2002 .

[54]  Jeff Dike,et al.  User-mode Linux , 2006, Annual Linux Showcase & Conference.

[55]  Andrea Westerinen,et al.  Terminology for Policy-Based Management , 2001, RFC.

[56]  Geoffrey H. Kuenning,et al.  Replication Requirements in Mobile Environments , 2001, Mob. Networks Appl..

[57]  Michael J. Carey,et al.  The Propel Distributed Services Platform , 2001, VLDB.

[58]  Ian T. Foster,et al.  Grid information services for distributed resource sharing , 2001, Proceedings 10th IEEE International Symposium on High Performance Distributed Computing.

[59]  S. Tuecke,et al.  Condor-G: A Computation Management Agent for Multi-Institutional Grids , 2001, Proceedings 10th IEEE International Symposium on High Performance Distributed Computing.

[60]  Stephen F. Bush,et al.  Predicting and controlling resource usage in a heterogeneous active network , 2001, Proceedings Third Annual International Workshop on Active Middleware Services.

[61]  A. Ryman Simple object access protocol (SOAP) and Web services , 2001, ICSE 2001.

[62]  Ben Immers,et al.  Time-of-day modeling using quasi-dynamic equilibrium assignement approach , 2001 .

[63]  Tim Owen,et al.  SafetyNet: A language-based approach to programmable networks , 2001, Comput. Networks.

[64]  Brian Tierney,et al.  Applied techniques for high bandwidth data transfers across wide area networks , 2001 .

[65]  Ian T. Foster,et al.  The anatomy of the grid: enabling scalable virtual organizations , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[66]  Emil C. Lupu,et al.  The Ponder Policy Specification Language , 2001, POLICY.

[67]  M. Sawhney,et al.  Where value lives in a networked world. , 2001, Harvard business review.

[68]  David P. Anderson,et al.  SETI@home-massively distributed computing for SETI , 2001, Comput. Sci. Eng..

[69]  Li Gong Project JXTA: A Technology Overview , 2001 .

[70]  J. Ritter Why Gnutella Can't Scale. No, Really , 2001 .

[71]  Evangelos P. Markatos,et al.  DIVISOR: DIstributed VIdeo Server fOr stReaming , 2001 .

[72]  Leendert van Doorn,et al.  A secure java TM virtual machine , 2000 .

[73]  Rajesh Raman,et al.  Resource management through multilateral matchmaking , 2000, Proceedings the Ninth International Symposium on High-Performance Distributed Computing.

[74]  Jeffrey O. Kephart,et al.  Dynamic pricing by software agents , 2000, Comput. Networks.

[75]  David Abramson,et al.  Nimrod/G: an architecture for a resource management and scheduling system in a global computational grid , 2000, Proceedings Fourth International Conference/Exhibition on High Performance Computing in the Asia-Pacific Region.

[76]  Leendert van Doorn,et al.  A Secure Java Virtual Machine , 2000, USENIX Security Symposium.

[77]  Ami Marowka,et al.  The GRID: Blueprint for a New Computing Infrastructure , 2000, Parallel Distributed Comput. Pract..

[78]  Germán S. Goldszmidt The Océano Project - A Multi-Domain Cluster for a Computing Utility , 2000, CLUSTER.

[79]  Mustaque Ahamad,et al.  Generalized Role-Based Access Control for Securing Future Applications , 2000 .

[80]  David Mazières,et al.  Separating key management from file system security , 1999, SOSP.

[81]  Emil C. Lupu,et al.  Conflicts in Policy-Based Distributed Systems Management , 1999, IEEE Trans. Software Eng..

[82]  Ian T. Foster,et al.  Resource co-allocation in computational grids , 1999, Proceedings. The Eighth International Symposium on High Performance Distributed Computing (Cat. No.99TH8469).

[83]  Bob Briscoe,et al.  A Dynamic Pricing Framework to support a Scalable, Usage-Based Charging Model for Packet-Switched Networks , 1999, IWAN.

[84]  Brian Walters,et al.  VMware Virtual Platform , 1999 .

[85]  Spencer Shepler NFS Version 4 Design Considerations , 1999, RFC.

[86]  Klara Nahrstedt,et al.  A distributed resource management architecture that supports advance reservations and co-allocation , 1999, 1999 Seventh International Workshop on Quality of Service. IWQoS'99. (Cat. No.98EX354).

[87]  David Wetherall,et al.  ANTS: Network Services Without the Red Tape , 1999, Computer.

[88]  Richard Mortier,et al.  An economic approach to adaptive resource management , 1999, Proceedings of the Seventh Workshop on Hot Topics in Operating Systems.

[89]  Christopher Allen,et al.  The TLS Protocol Version 1.0 , 1999, RFC.

[90]  Hugo Krawczyk,et al.  A Security Architecture for the Internet Protocol , 1999, IBM Syst. J..

[91]  Steven McCanne,et al.  An active service framework and its application to real-time multimedia transcoding , 1998, SIGCOMM '98.

[92]  Carl A. Gunter,et al.  PLAN: a packet language for active networks , 1998, ICFP '98.

[93]  Rajesh Raman,et al.  Matchmaking: distributed resource management for high throughput computing , 1998, Proceedings. The Seventh International Symposium on High Performance Distributed Computing (Cat. No.98TB100244).

[94]  Jean Bacon,et al.  Access control in an open distributed environment , 1998, Proceedings. 1998 IEEE Symposium on Security and Privacy (Cat. No.98CB36186).

[95]  Daniela Rus,et al.  Market-based resource control for mobile agents , 1998, AGENTS '98.

[96]  John V. Guttag,et al.  ANTS: a toolkit for building and dynamically deploying network protocols , 1998, 1998 IEEE Open Architectures and Network Programming.

[97]  Warren Smith,et al.  A Resource Management Architecture for Metacomputing Systems , 1998, JSSPP.

[98]  R. Peffer What is to Be Distributed , 1998 .

[99]  Franco Travostino,et al.  Towards a Resource-safe Java for Service Guarantees in Uncooperative Environments , 1998 .

[100]  Ladislau Bölöni,et al.  An Alternative Model for Scheduling on a Computational Grid , 1998 .

[101]  Godmar Back Patrick Tullmann Leigh Stoller Wilson C. Hsie Lepreau Java Operating Systems : Design and Implementation , 1998 .

[102]  Tim Howes,et al.  Lightweight Directory Access Protocol (v3) , 1997, RFC.

[103]  Thomas Kunz,et al.  Services Supporting Management of Distributed Applications and Systems , 1997, IBM Syst. J..

[104]  Mahadev Satyanarayanan,et al.  Agile application-aware adaptation for mobility , 1997, SOSP.

[105]  Warren Smith,et al.  A directory service for configuring high-performance distributed computations , 1997, Proceedings. The Sixth IEEE International Symposium on High Performance Distributed Computing (Cat. No.97TB100183).

[106]  Ian T. Foster,et al.  Globus: a Metacomputing Infrastructure Toolkit , 1997, Int. J. High Perform. Comput. Appl..

[107]  Ken Moody,et al.  An open architecture for secure interworking services , 1996, Proceedings of 17th International Conference on Distributed Computing Systems.

[108]  C. M. Sperberg-McQueen,et al.  Extensible Markup Language (XML) , 1997, World Wide Web J..

[109]  Miron Livny,et al.  Mechanisms for High Throughput Computing , 1997 .

[110]  George C. Necula,et al.  Proof-carrying code , 1997, POPL '97.

[111]  Emil C. Lupu,et al.  A policy based role framework for access control , 1996, RBAC '95.

[112]  Tatu Ylonen,et al.  SSH: secure login connections over the internet , 1996 .

[113]  Aaron Kershenbaum,et al.  Mobile Agents: Are They a Good Idea? , 1996, Mobile Object Systems.

[114]  Mahadev Satyanarayanan,et al.  An empirical study of a wide-area distributed file system , 1996, TOCS.

[115]  Roger Riggs,et al.  A Distributed Object Model for the Java System , 1996, Comput. Syst..

[116]  Brian N. Bershad,et al.  Extensibility safety and performance in the SPIN operating system , 1995, SOSP.

[117]  Randall J. Atkinson,et al.  Security Architecture for the Internet Protocol , 1995, RFC.

[118]  Gordon S. Blair,et al.  A Network Emulator to Support the Development of Adaptive Applications , 1995, Symposium on Mobile and Location-Independent Computing.

[119]  Tim Howes,et al.  Lightweight Directory Access Protocol , 1995, RFC.

[120]  Klaus R. Dittrich,et al.  Argos - A Configurable Access Control System for Interoperable Environments , 1995, DBSec.

[121]  Ken Thompson,et al.  Plan 9 from Bell Labs , 1995 .

[122]  Theodore Y. Ts'o,et al.  Kerberos: an authentication service for computer networks , 1994, IEEE Communications Magazine.

[123]  Sylvia L. Osborn,et al.  Access Rights Administration in Role-Based Security Systems , 1994, DBSec.

[124]  Glenn Mansfield Keeni,et al.  Charting Networks in the X.500 Directory , 1994, RFC.

[125]  Peter Honeyman,et al.  Disconnected Operation for AFS , 1993, Symposium on Mobile and Location-Independent Computing.

[126]  Ken Thompson,et al.  The use of name spaces in Plan 9 , 1993, OPSR.

[127]  Ken Thompson,et al.  The use of name spaces in plan 9 , 1992, EW 5.

[128]  Al Geist,et al.  Network-based concurrent computing on the PVM system , 1992, Concurr. Pract. Exp..

[129]  Mahadev Satyanarayanan,et al.  The Influence of Scale on Distributed File System Design , 1992, IEEE Trans. Software Eng..

[130]  James J. Kistler,et al.  Disconnected operation in the Coda file system , 1991, SOSP '91.

[131]  Richard F. Freund,et al.  Superconcurrency: A Form of Distributed Heterogeneous Supercomputing , 1991 .

[132]  Vaidy S. Sunderam,et al.  PVM: A Framework for Parallel Distributed Computing , 1990, Concurr. Pract. Exp..

[133]  Ravishankar K. Iyer,et al.  Predictability of Process Resource Usage: A Measurement-Based Study on UNIX , 1989, IEEE Trans. Software Eng..

[134]  R. F. Freund,et al.  Optimal selection theory for superconcurrency , 1989, Proceedings of the 1989 ACM/IEEE Conference on Supercomputing (Supercomputing '89).

[135]  Amos Fiat,et al.  Untraceable Electronic Cash , 1990, CRYPTO.

[136]  Miron Livny,et al.  Condor-a hunter of idle workstations , 1988, [1988] Proceedings. The 8th International Conference on Distributed.

[137]  Butler Lampson,et al.  Personal distributed computing: the Alto and Ethernet software , 1986, HPW '86.

[138]  Andrew Birrell,et al.  Implementing remote procedure calls , 1984, TOCS.

[139]  A. J. Herbert The Cambridge Distributed Computing System , 1983, Advanced Course: Local Area Networks.

[140]  John F. Shoch,et al.  The “worm” programs—early experience with a distributed computation , 1982, CACM.

[141]  Stephen T. Kent Protecting externally supplied software in small computers , 1980 .

[142]  Gordon M. Bull The Dartmouth Time-Sharing System , 1980 .

[143]  Shahid H. Bokhari,et al.  Control of Distributed Processes , 1978, Computer.

[144]  Philip H. Enslow What is a "Distributed" Data Processing System? , 1978, Computer.

[145]  Jim Gray,et al.  Notes on Data Base Operating Systems , 1978, Advanced Course: Operating Systems.

[146]  David C. Walden,et al.  The ARPA Network Design Decisions , 1977, Comput. Networks.

[147]  Whitfield Diffie,et al.  New Directions in Cryptography , 1976, IEEE Trans. Inf. Theory.

[148]  Butler W. Lampson,et al.  Crash Recovery in a Distributed Data Storage System , 1981 .

[149]  Jerome H. Saltzer,et al.  The protection of information in computer systems , 1975, Proc. IEEE.

[150]  L. Hurwicz The Design of Mechanisms for Resource Allocation , 1973 .

[151]  Richard P. Parmelee,et al.  Virtual Storage and Virtual Machine Concepts , 1972, IBM Syst. J..

[152]  Lee L. Selwyn,et al.  Considerations for computer utility pricing policies , 1968, ACM National Conference.

[153]  Michael J. Flynn,et al.  Very high-speed computing systems , 1966 .

[154]  L. W. Comeau,et al.  A VIRTUAL MACHINE SYSTEM FOR THE 360/40 , 1966 .

[155]  R. M. Fano,et al.  The MAC system: the computer utility approach , 1965, IEEE Spectrum.

[156]  Jack B. Dennis,et al.  A multiuser computation facility for education and research , 1964, CACM.

[157]  Robert C. Daley,et al.  An experimental time-sharing system , 1962, AIEE-IRE '62 (Spring).