Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M

The Dynamic Distributed Dimensional Data Model (D4M) library implements associative arrays in a variety of languages (Python, Julia, and Matlab/Octave) and provides a lightweight in-memory database implementation of hypersparse arrays that are ideal for analyzing many types of network data. D4M relies on associative arrays which combine properties of spreadsheets, databases, matrices, graphs, and networks, while providing rigorous mathematical guarantees, such as linearity. Streaming updates of D4M associative arrays put enormous pressure on the memory hierarchy. This work describes the design and performance optimization of an implementation of hierarchical associative arrays that reduces memory pressure and dramatically increases the update rate into an associative array. The parameters of hierarchical associative arrays rely on controlling the number of entries in each level in the hierarchy before an update is cascaded. The parameters are easily tunable to achieve optimal performance for a variety of applications. Hierarchical arrays achieve over 40,000 updates per second in a single instance. Scaling to 34,000 instances of hierarchical D4M associative arrays on 1,100 server nodes on the MIT SuperCloud achieved a sustained update rate of 1,900,000,000 updates per second. This capability allows the MIT SuperCloud to analyze extremely large streaming network data sets.

[1]  Michael Stonebraker,et al.  A Demonstration of SciDB: A Science-Oriented DBMS , 2009, Proc. VLDB Endow..

[2]  Bradley A. Mason.  Tropical Algebra, Graph Theory, & Foreign Exchange Arbitrage , 2016 .

[3]  Paul Klemperer,et al.  Understanding Preferences: 'Demand Types', and The Existence of Equilibrium with Indivisibilities , 2018 .

[4]  Jeremy Kepner,et al.  Dynamic distributed dimensional data model (D4M) database and computation system , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[5]  E. F. CODD,et al.  A relational model of data for large shared data banks , 1970, CACM.

[6]  Jeremy Kepner,et al.  Lustre, hadoop, accumulo , 2015, 2015 IEEE High Performance Extreme Computing Conference (HPEC).

[7]  Paul J. Werbos,et al.  The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting , 1994 .

[8]  Michael Stonebraker,et al.  The design and implementation of INGRES , 1976, TODS.

[9]  Prashant Malik,et al.  Cassandra: a decentralized structured storage system , 2010, OPSR.

[10]  Michael Stonebraker,et al.  The VoltDB Main Memory DBMS , 2013, IEEE Data Eng. Bull..

[11]  Jeremy Kepner,et al.  Achieving 100,000,000 database inserts per second using Accumulo and D4M , 2014, 2014 IEEE High Performance Extreme Computing Conference (HPEC).

[12]  David Maier,et al.  The Theory of Relational Databases , 1983 .

[13]  Werner Vogels,et al.  Dynamo: amazon's highly available key-value store , 2007, SOSP.

[14]  P. Klemperer.  The Product-Mix Auction: A New Auction Design for Differentiated Goods , 2010 .

[15]  Sivasankaran Rajamanickam,et al.  Fast Triangle Counting Using Cilk , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[16]  Jeremy Kepner,et al.  Polystore mathematics of relational algebra , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[17]  José E. Moreira,et al.  Enabling massive deep neural networks with the GraphBLAS , 2017, 2017 IEEE High Performance Extreme Computing Conference (HPEC).

[18]  Luiz André Barroso,et al.  Web Search for a Planet: The Google Cluster Architecture , 2003, IEEE Micro.

[19]  David Elliot Shaw.  A Relational Database Machine Architecture , 1980, CAW '80.

[20]  Lars George,et al.  HBase: The Definitive Guide , 2011 .

[21]  William Song,et al.  Static graph challenge: Subgraph isomorphism , 2017, 2017 IEEE High Performance Extreme Computing Conference (HPEC).

[22]  Srivatsa S. Bhat,et al.  Designing multicore scalable filesystems with durability and crash consistency , 2017 .

[23]  Jeremy Kepner,et al.  Benchmarking SciDB data import on HPC systems , 2016, 2016 IEEE High Performance Extreme Computing Conference (HPEC).

[24]  Michael Stonebraker,et al.  The Case for Shared Nothing , 1985, HPTS.

[25]  Wilson C. Hsieh,et al.  Spanner , 2012, OSDI.

[26]  Austin T. Clements,et al.  The scalable commutativity rule: designing scalable software for multicore processors , 2013, SOSP.

[27]  Ranjan Sen,et al.  Benchmarking Apache Accumulo BigData Distributed Table Store Using Its Continuous Test Suite , 2013, 2013 IEEE International Congress on Big Data.

[28]  Jack J. Dongarra,et al.  The LINPACK Benchmark: past, present and future , 2003, Concurr. Comput. Pract. Exp..

[29]  John Tran,et al.  cuDNN: Efficient Primitives for Deep Learning , 2014, ArXiv.

[30]  Jeremy Kepner,et al.  Graphulo implementation of server-side sparse matrix multiply in the Accumulo database , 2015, 2015 IEEE High Performance Extreme Computing Conference (HPEC).

[31]  Jeremy Kepner,et al.  Hyperscaling Internet Graph Analysis with D4M on the MIT SuperCloud , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[32]  William Song,et al.  Streaming graph challenge: Stochastic block partition , 2017, 2017 IEEE High Performance Extreme Computing Conference (HPEC).

[33]  H. Howie Huang,et al.  High-Performance Triangle Counting on GPUs , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[34]  Michael Wall,et al.  Accumulo: Application Development, Table Design, and Best Practices , 2015 .

[35]  Carlo Curino,et al.  Schism , 2010, Proc. VLDB Endow..

[36]  Michael Stonebraker,et al.  H-store: a high-performance, distributed main memory transaction processing system , 2008, Proc. VLDB Endow..

[37]  Michael Stonebraker,et al.  C-Store: A Column-oriented DBMS , 2005, VLDB.

[38]  Grayce M. Booth.  Distributed information systems , 1976, AFIPS '76.

[39]  Roger Pearce,et al.  K-truss decomposition for Scale-Free Graphs at Scale in Distributed Memory , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[40]  Ramez Elmasri,et al.  Fundamentals of Database Systems , 1989 .

[41]  David A. Bader,et al.  Hornet: An Efficient Data Structure for Dynamic Sparse Graphs and Matrices on GPUs , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[42]  Jeremy Kepner,et al.  Graphulo: Linear Algebra Graph Kernels for NoSQL Databases , 2015, 2015 IEEE International Parallel and Distributed Processing Symposium Workshop.

[43]  Mauro Bisson,et al.  Update on Static Graph Challenge on GPU , 2018, 2018 IEEE High Performance extreme Computing Conference (HPEC).

[44]  C. J. Date,et al.  A Guide to the SQL Standard: A User's Guide to the Standard Relational Language SQL , 1993 .

[45]  Martin Fodslette Meiller.  A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning , 1993 .

[46]  Marco Minutoli,et al.  High-Performance Data Analytics Beyond the Relational and Graph Data Models with GEMS , 2017, 2017 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW).

[47]  Carlo Curino,et al.  Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems , 2012, SIGMOD Conference.