Meta-Learning Neural Bloom Filters

There has been a recent trend in training neural networks to replace data structures that have been crafted by hand, with an aim for faster execution, better accuracy, or greater compression. In this setting, a neural data structure is instantiated by training a network over many epochs of its inputs until convergence. In applications where inputs arrive at high throughput, or are ephemeral, training a network from scratch is not practical. This motivates the need for few-shot neural data structures. In this paper we explore the learning of approximate set membership over a set of data in one-shot via meta-learning. We propose a novel memory architecture, the Neural Bloom Filter, which is able to achieve significant compression gains over classical Bloom Filters and existing memory-augmented neural networks.

[1]  Pascal Vincent,et al.  Hierarchical Memory Networks , 2016, ArXiv.

[2]  Tim Kraska,et al.  The Case for Learned Index Structures , 2018 .

[3]  Larry Carter,et al.  Exact and approximate membership testers , 1978, STOC.

[4]  S. Muthukrishnan,et al.  Data streams: algorithms and applications , 2005, SODA '03.

[5]  Bin Fan,et al.  Cuckoo Filter: Practically Better Than Bloom , 2014, CoNEXT.

[6]  Bernard Chazelle,et al.  The Bloomier filter: an efficient data structure for static support lookup tables , 2004, SODA '04.

[7]  Li Fan,et al.  Summary cache: a scalable wide-area web cache sharing protocol , 2000, TNET.

[8]  Yusheng Ji,et al.  Improved Weighted Bloom Filter and Space Lower Bound Analysis of Algorithms for Approximated Membership Querying , 2015, DASFAA.

[9]  Mahmood Ahmadi,et al.  Bloom filter applications in network security: A state-of-the-art survey , 2013, Comput. Networks.

[10]  Philip Sterne,et al.  Efficient and robust associative memory from a generalized Bloom filter , 2012, Biological Cybernetics.

[11]  Michael Mitzenmacher,et al.  A Model for Learned Bloom Filters and Related Structures , 2018, ArXiv.

[12]  Alex Graves,et al.  The Kanerva Machine: A Generative Distributed Memory , 2018, ICLR.

[13]  Michael Mitzenmacher,et al.  Compressed bloom filters , 2001, PODC '01.

[14]  Sergio Gomez Colmenarejo,et al.  Hybrid computing using a neural network with dynamic external memory , 2016, Nature.

[15]  Geoffrey E. Hinton,et al.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , 2017, ICLR.

[16]  Jie Gao,et al.  Weighted Bloom filter , 2006, 2006 IEEE International Symposium on Information Theory.

[17]  Ravi Kumar,et al.  An information statistics approach to data stream and communication complexity , 2004, J. Comput. Syst. Sci..

[18]  Chris Cummins,et al.  End-to-End Deep Learning of Optimization Heuristics , 2017, 2017 26th International Conference on Parallel Architectures and Compilation Techniques (PACT).

[19]  Martin Dietzfelbinger,et al.  Succinct Data Structures for Retrieval and Approximate Membership , 2008, ICALP.

[20]  Andrew Chi-Chih Yao,et al.  Some complexity questions related to distributive computing(Preliminary Report) , 1979, STOC.

[21]  Panagiotis Manolios,et al.  Bloom Filters in Probabilistic Verification , 2004, FMCAD.

[22]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[23]  Samy Bengio,et al.  Device Placement Optimization with Reinforcement Learning , 2017, ICML.

[24]  David A. Patterson,et al.  In-datacenter performance analysis of a tensor processing unit , 2017, 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA).

[25]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[26]  Alex Graves,et al.  Neural Turing Machines , 2014, ArXiv.

[27]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[28]  Bartunov Sergey,et al.  Meta-Learning with Memory-Augmented Neural Networks , 2016 .

[29]  Nicole Immorlica,et al.  Locality-sensitive hashing scheme based on p-stable distributions , 2004, SCG '04.

[30]  Rafal Bogacz,et al.  Comparison of computational models of familiarity discrimination in the perirhinal cortex , 2003, Hippocampus.

[31]  Abhishek Kumar,et al.  A New Design of Bloom Filter for Packet Inspection Speedup , 2007, IEEE GLOBECOM 2007 - IEEE Global Telecommunications Conference.

[32]  Shigang Chen,et al.  One memory access bloom filters and their generalization , 2011, 2011 Proceedings IEEE INFOCOM.

[33]  Aurko Roy,et al.  Learning to Remember Rare Events , 2017, ICLR.

[34]  Michael Mitzenmacher.  Optimizing Learned Bloom Filters by Sandwiching , 2018, ArXiv.

[35]  Yan Wu,et al.  Learning Attractor Dynamics for Generative Memory , 2018, NeurIPS.

[36]  Jason Weston,et al.  End-To-End Memory Networks , 2015, NIPS.

[37]  Alex Graves,et al.  Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes , 2016, NIPS.