The design and implementation of declarative networks

In this dissertation, we present the design and implementation of declarative networks. Declarative networking proposes the use of a declarative query language for specifying and implementing network protocols, and employs a dataflow framework at runtime for com munication and maintenance of network state. The primary goal of declarative networking is to greatly simplify the process of specifying, implementing, deploying and evolving a network design. In addition, declarative networking serves as an important step towards an extensible, evolvable network architecture that can support flexible, secure and efficient deployment of new network protocols. Our main contributions are as follows. First, we formally define the Network Datalog (NDlog) language based on extensions to the Datalog recursive query language, and propose NDlog as a Domain Specific Language for programming network protocols. We demonstrate that NDlog can be used to express a large variety of network protocols in a handful of lines of program code, resulting in orders of magnitude reduction in code size. For example, the Chord overlay network can be specified in 48 NDlog rules. In addition, the core of the language (Datalog) has polynomial complexity, and our NDlog extensions can be statically analyzed for termination using standard analysis techniques. Second, to validate the design of NDlog, we present our implementation of P2, which is a full-fledged declarative networking system that compiles NDlog and executes it via a dataflow engine inspired by the Click modular router. We experimentally evaluate the P2 system on hundreds of distributed machines. The P2 system is publicly available for download and has been used in research projects at a number of institutions. Third, based on our experiences implementing declarative networks, we explore a wide variety of database research issues that are important for the practical realization of declarative networks. These include pipelined execution of distributed recursive queries, reason ing about query semantics based on the well-known distributed systems notion of "eventual consistency", incorporating the notion of soft-state into the logical framework of NDlog, and the use of query optimizations to improve the performance of network protocols.

[1]  Ion Stoica,et al.  The Case for a Hybrid P2P Search Infrastructure , 2004, IPTPS.

[2]  S. Sudarshan,et al.  Aggregation and Relevance in Deductive Databases , 1991, VLDB.

[3]  Ion Stoica,et al.  Analyzing P2P Overlays with Recursive Queries , 2003 .

[4]  Diego Calvanese,et al.  Decidable containment of recursive queries , 2003, Theor. Comput. Sci..

[5]  Miguel Castro,et al.  Secure routing for structured peer-to-peer overlay networks , 2002, OSDI '02.

[6]  Monica S. Lam,et al.  Cloning-based context-sensitive pointer alias analysis using binary decision diagrams , 2004, PLDI '04.

[7]  Raghu Ramakrishnan,et al.  Review - Magic Sets and Other Strange Ways to Implement Logic Programs , 1999, ACM SIGMOD Digit. Rev..

[8]  Carl A. Gunter,et al.  PLAN: a packet language for active networks , 1998, ICFP '98.

[9]  Thomas Anderson,et al.  Overcoming Barriers to Disruptive Innovation in Networking , 2005 .

[10]  Pavlin Radoslavov,et al.  Designing extensible IP router software , 2005, NSDI.

[11]  Scott Shenker,et al.  The Architecture of PIER: an Internet-Scale Query Processor , 2005, CIDR.

[12]  EDDIE KOHLER,et al.  The click modular router , 2000, TOCS.

[13]  Xiaowei Yang,et al.  NIRA: a new Internet routing architecture , 2003, FDNA '03.

[14]  Laurent Vieille,et al.  Recursive Axioms in Deductive Databases: The Query/Subquery Approach , 1986, Expert Database Conf..

[15]  Jon Crowcroft,et al.  Core Based Trees (CBT) An Architecture for Scalable Inter-Domain Multicast Routing , 1993, SIGCOMM 1993.

[16]  Marvin Theimer,et al.  Managing update conflicts in Bayou, a weakly connected replicated storage system , 1995, SOSP.

[17]  Dan Suciu,et al.  Dynamically distributed query evaluation , 2001, PODS.

[18]  François Bancilhon,et al.  Naive Evaluation of Recursively Defined Relations , 1986, On Knowledge Base Management Systems.

[19]  Srinivasan Keshav,et al.  Proceedings of the ACM SIGCOMM workshop on Future directions in network architecture , 2004, SIGCOMM 2004.

[20]  David E. Culler,et al.  Monsoon: an explicit token-store architecture , 1998, ISCA '98.

[21]  Ion Stoica,et al.  Implementing declarative overlays , 2005, SOSP '05.

[22]  Patrick Valduriez,et al.  Principles of distributed database systems (2nd ed.) , 1999 .

[23]  Srinivasan Seshan,et al.  A case for end system multicast , 2002, IEEE J. Sel. Areas Commun..

[24]  Zygmunt J. Haas,et al.  A new routing protocol for the reconfigurable wireless networks , 1997, Proceedings of ICUPC 97 - 6th International Conference on Universal Personal Communications.

[25]  Goetz Graefe,et al.  Encapsulation of parallelism in the Volcano query processing system , 1990, SIGMOD '90.

[26]  Hamid Pirahesh,et al.  Cost-based optimization for magic: algebra and implementation , 1996, SIGMOD '96.

[27]  Trevor Jim,et al.  SD3: a trust management system with certified evaluation , 2001, Proceedings 2001 IEEE Symposium on Security and Privacy. S&P 2001.

[28]  Martín Abadi,et al.  Logic in access control , 2003, 18th Annual IEEE Symposium of Logic in Computer Science, 2003. Proceedings..

[29]  Nick Feamster,et al.  Towards a logic for wide-area Internet routing , 2003, FDNA '03.

[30]  Angelos D. Keromytis,et al.  Proceedings of the Freenix Track: 2003 Usenix Annual Technical Conference Secure and Flexible Global File Sharing , 2022 .

[31]  John DeTreville,et al.  Binder, a logic-based security language , 2002, Proceedings 2002 IEEE Symposium on Security and Privacy.

[32]  Carlo Zaniolo,et al.  The deductive database system [Lscr ][Dscr ][Lscr ]++ , 2002, Theory and Practice of Logic Programming.

[33]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[34]  Robert Tappan Morris,et al.  Designing a DHT for Low Latency and High Throughput , 2004, NSDI.

[35]  Ravi Krishnamurthy,et al.  A Framework for Testing Safety and Effective Computability , 1996, J. Comput. Syst. Sci..

[36]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM '01.

[37]  David J. DeWitt,et al.  GAMMA - A High Performance Dataflow Database Machine , 1986, VLDB.

[38]  Joseph M. Hellerstein,et al.  Eddies: continuously adaptive query processing , 2000, SIGMOD '00.

[39]  Steven McCanne,et al.  A model, analysis, and protocol framework for soft state-based communication , 1999, SIGCOMM '99.

[40]  A. Rowstron,et al.  Scalable, decentralized object location and routing for large-scale peer-to-peer systems , 2001 .

[41]  Ion Stoica,et al.  Declarative networking: language, execution and optimization , 2006, SIGMOD Conference.

[42]  Joseph M. Hellerstein,et al.  A need for componentized transport protocols , 2005, SOSP '05.

[43]  Arie van Deursen,et al.  Domain-specific languages: an annotated bibliography , 2000, SIGP.

[44]  Amin Vahdat,et al.  MACEDON: Methodology for Automatically Creating, Evaluating, and Designing Overlay Networks , 2004, NSDI.

[45]  Frederick Reiss,et al.  TelegraphCQ: Continuous Dataflow Processing for an Uncertain World , 2003, CIDR.

[46]  V. S. Subrahmanian,et al.  Maintaining views incrementally , 1993, SIGMOD Conference.

[47]  Dawson R. Engler,et al.  Model Checking Large Network Protocol Implementations , 2004, NSDI.

[48]  Joan Feigenbaum,et al.  The Role of Trust Management in Distributed Systems Security , 2001, Secure Internet Programming.

[49]  Scott Shenker,et al.  Overcoming the Internet impasse through virtualization , 2005, Computer.

[50]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[51]  F. Leighton,et al.  Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes , 1991 .

[52]  Scott Shenker,et al.  Routing as a Service , 2006 .

[53]  Andrew W. Appel,et al.  MulVAL: A Logic-based Network Security Analyzer , 2005, USENIX Security Symposium.

[54]  Krishna P. Gummadi,et al.  The impact of DHT routing geometry on resilience and proximity , 2003, SIGCOMM '03.

[55]  Michael Stonebraker,et al.  Mariposa: a wide-area distributed database system , 1996, The VLDB Journal.

[56]  Nick Feamster,et al.  The case for separating routing from routers , 2004, FDNA '04.

[57]  Emin Gün Sirer,et al.  Cost-aware resource management for decentralized internet services , 2007 .

[58]  Robbert van Renesse,et al.  Scalable Management and Data Mining Using Astrolabe , 2002, IPTPS.

[59]  Kotagiri Ramamohanarao,et al.  A Generalization of the Differential Approach to Recursive Query Evaluation , 1987, J. Log. Program..

[60]  Srinivasan Seshan,et al.  IrisNet: An Architecture for a Worldwide Sensor Web , 2003, IEEE Pervasive Comput..

[61]  Readings in Database Systems, Third Edition , 1998 .

[62]  Ion Stoica,et al.  Declarative routing: extensible routing with declarative queries , 2005, SIGCOMM '05.

[63]  Stefano Ceri,et al.  A survey of parallel execution strategies for transitive closure and logic programs , 1993, Distributed and Parallel Databases.

[64]  Patrick Valduriez,et al.  Principles of Distributed Database Systems , 1990 .

[65]  Boon Thau Loo,et al.  Distributed Web Crawling over DHTs , 2004 .

[66]  Jeffrey D. Ullman,et al.  A Survey of Research in Deductive Database Systems , 1995 .

[67]  Sergio Greco,et al.  Pushing extrema aggregates to optimize logic queries , 2002, Inf. Syst..

[68]  David Chu,et al.  Entirely declarative sensor network systems , 2006, VLDB.

[69]  Ben Y. Zhao,et al.  An Infrastructure for Fault-tolerant Wide-area Location and Routing , 2001 .

[70]  Ion Stoica,et al.  Customizable Routing with Declarative Queries , 2004 .

[71]  Scott Shenker,et al.  Enhancing P2P File-Sharing with an Internet-Scale Query Processor , 2004, Very Large Data Bases Conference.

[72]  Bassam Halabi,et al.  Internet Routing Architectures , 1997 .

[73]  Lei Gao,et al.  PRACTI Replication , 2006, NSDI.

[74]  Peter M. G. Apers,et al.  Pipelining in query execution , 1990, Proceedings. PARBASE-90: International Conference on Databases, Parallel Architectures, and Their Applications.

[75]  Leslie Lamport,et al.  The temporal logic of actions , 1994, TOPL.

[76]  David D. Clark,et al.  The design philosophy of the DARPA internet protocols , 1988, SIGCOMM '88.

[77]  Jeffrey D. Ullman,et al.  Assigning an Appropriate Meaning to Database Logic With Negation , 1994 .

[78]  Dieter Richter.  Using formal description techniques, an introduction to ESTELLE, LOTOS and SDL: by Kenneth J. Turner, ed., 1992. XXVIII + 431 pages, hardcover. ISBN 0-471-93455-0 , 1993 .

[79]  Stephen E. Deering,et al.  Multicast routing in datagram internetworks and extended LANs , 1990, TOCS.

[80]  David J. DeWitt,et al.  Parallel database systems: the future of high performance database systems , 1992, CACM.

[81]  Brian N. Bershad,et al.  Extensibility safety and performance in the SPIN operating system , 1995, SOSP.

[82]  Patrick Valduriez,et al.  Principles of Distributed Database Systems, Second Edition , 1999 .

[83]  John Kubiatowicz,et al.  Handling churn in a DHT , 2004 .

[84]  Gurmeet Singh Manku,et al.  Symphony: Distributed Hashing in a Small World , 2003, USENIX Symposium on Internet Technologies and Systems.

[85]  Serge Abiteboul,et al.  Foundations of Databases , 1994 .

[86]  Peter Sewell,et al.  Cassandra: distributed access control policies with tunable expressiveness , 2004, Proceedings. Fifth IEEE International Workshop on Policies for Distributed Systems and Networks, 2004. POLICY 2004..

[87]  Catriel Beeri,et al.  On the power of magic , 1987, J. Log. Program..

[88]  Martín Abadi.  On Access Control, Data Integration, and Their Languages , 2004 .

[89]  G.J. Minden,et al.  A survey of active network research , 1997, IEEE Communications Magazine.

[90]  David A. Maltz,et al.  Dynamic Source Routing in Ad Hoc Wireless Networks , 1994, Mobidata.

[91]  Arthur H. Veen,et al.  Dataflow machine architecture , 1986, CSUR.

[92]  Serge Abiteboul,et al.  Diagnosis of asynchronous discrete event systems: datalog to the rescue! , 2005, PODS.

[93]  Michael Stonebraker,et al.  Inclusion of new types in relational data base systems , 1986, 1986 IEEE Second International Conference on Data Engineering.

[94]  Jennifer Widom,et al.  The TSIMMIS Project: Integration of Heterogeneous Information Sources , 1994, IPSJ.

[95]  Joan Feigenbaum,et al.  Delegation logic: A logic-based approach to distributed authorization , 2003, TSEC.

[96]  Larry L. Peterson,et al.  Making paths explicit in the Scout operating system , 1996, OSDI '96.

[97]  Kenneth A. Ross,et al.  Efficient Incremental Evaluation of Queries with Aggregation , 1994, ILPS.

[98]  Bruce S. Davie,et al.  Computer Networks: A Systems Approach , 1996 .

[99]  Olga Papaemmanouil,et al.  Extensible optimization in overlay dissemination trees , 2006, SIGMOD Conference.

[100]  Paul D. Amer,et al.  A success story of formal description techniques: Estelle specification and test generation for MIL-STD 188-220 , 2000, Comput. Commun..

[101]  David A. Bell,et al.  An experimental performance study of a pipelined recursive query processing strategy , 1990, DPDS '90.

[102]  David R. Karger,et al.  On the Feasibility of Peer-to-Peer Web Indexing and Search , 2003, IPTPS.

[103]  William F. Clocksin,et al.  Programming in Prolog , 1981, Springer Berlin Heidelberg.

[104]  Christos Faloutsos,et al.  ANF: a fast and scalable tool for data mining in massive graphs , 2002, KDD.

[105]  Atul Singh,et al.  Using queries for distributed monitoring and forensics , 2006, EuroSys.

[106]  Emin Gün Sirer,et al.  SHARP: a hybrid adaptive routing protocol for mobile ad hoc networks , 2003, MobiHoc '03.

[107]  Joseph M. Hellerstein,et al.  Induced Churn as Shelter from Routing-Table Poisoning , 2006, NDSS.

[108]  Joseph M. Hellerstein,et al.  Finally, a Use for Componentized Transport Protocols , 2006 .

[109]  Nick Feamster,et al.  Correctness Properties for Internet Routing , 2005 .

[110]  Charles Consel,et al.  Safe and efficient active network programming , 1998, Proceedings Seventeenth IEEE Symposium on Reliable Distributed Systems (Cat. No.98CB36281).

[111]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[112]  Michael Stonebraker,et al.  Readings in Database Systems , 1988 .

[113]  Michael Stonebraker,et al.  Future Directions in DBMS Research , 1989 .

[114]  G. Weikum.  Querying the Internet with PIER , 2005 .

[115]  F. E..  A Relational Model of Data Large Shared Data Banks , 2000 .