Nanosecond Indexing of Graph Data With Hash Maps and VLists

We introduce a wait-free, multi-reader, single-writer, "kill -9" durable, indexing structure for in-memory social graph databases. This structure requires no communication from the readers back to the writer, allowing for trivial read scalability and isolation. We support online updates without compromising availability or read performance. Our structure supports looking up small subgraphs in 80 nanoseconds and a materialization rate of 12 nanoseconds per edge. Storage takes 7 bytes per edge per index and supports almost 1 million online writes per second.

[1]  Marko A. Rodriguez,et al.  Constructions from Dots and Lines , 2010, ArXiv.

[2]  Christos Faloutsos,et al.  PEGASUS: mining peta-scale graphs , 2011, Knowledge and Information Systems.

[3]  Gerhard Weikum,et al.  The RDF-3X engine for scalable management of RDF data , 2010, The VLDB Journal.

[4]  Andrew Carter,et al.  Moolle: Fan-out control for scalable distributed data stores , 2016, 2016 IEEE 32nd International Conference on Data Engineering (ICDE).

[5]  Jutta Degener,et al.  Optimizing schema-last tuple-store queries in graphd , 2010, SIGMOD Conference.

[6]  B. Briscoe,et al.  Metcalfe's law is wrong - communications networks increase in value as they add members-but by how much? , 2006, IEEE Spectrum.

[7]  Jan van Leeuwen,et al.  Worst-Case Optimal Insertion and Deletion Methods for Decomposable Searching Problems , 1981, Inf. Process. Lett..

[8]  Abraham Bernstein,et al.  Hexastore: sextuple indexing for semantic web data management , 2008, Proc. VLDB Endow..

[9]  Gang Hu,et al.  SQLGraph: An Efficient Relational-Based Property Graph Store , 2015, SIGMOD Conference.

[10]  Andrew Carter,et al.  Partial Update: Efficient Materialized View Maintenance in a Distributed Graph Database , 2018, 2018 IEEE 34th International Conference on Data Engineering (ICDE).

[11]  No License,et al.  Intel ® 64 and IA-32 Architectures Software Developer ’ s Manual Volume 3 A : System Programming Guide , Part 1 , 2006 .

[12]  Phil Bagwell,et al.  Fast Functional Lists, Hash-Lists, Deques, and Variable Length Arrays , 2002 .

[13]  Eddie Kohler,et al.  Cache craftiness for fast multicore key-value storage , 2012, EuroSys '12.

[14]  Daniel J. Abadi,et al.  Scalable SPARQL querying of large RDF graphs , 2011, Proc. VLDB Endow..

[15]  Aart J. C. Bik,et al.  Pregel: a system for large-scale graph processing , 2010, SIGMOD Conference.

[16]  Jack A. Orenstein,et al.  The ObjectStore database system , 1991, CACM.

[17]  Haixun Wang,et al.  Trinity: a distributed graph engine on a memory cloud , 2013, SIGMOD '13.

[18]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[19]  W. W. Peterson,et al.  Addressing for Random-Access Storage , 1957, IBM J. Res. Dev..

[20]  Hui Ding,et al.  TAO: Facebook's Distributed Data Store for the Social Graph , 2013, USENIX Annual Technical Conference.

[21]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[22]  Daniel J. Abadi,et al.  Scalable Semantic Web Data Management Using Vertical Partitioning , 2007, VLDB.

[23]  Phil Bagwell.  Fast Functional Lists , 2002, IFL.

[24]  Peter L. Montgomery,et al.  Division by invariant integers using multiplication , 1994, PLDI '94.