A comparison of vector symbolic architectures

Vector Symbolic Architectures (VSAs) combine a high-dimensional vector space with a set of carefully designed operators in order to perform symbolic computations with large numerical vectors. Major goals are the exploitation of their representational power and ability to deal with fuzziness and ambiguity. Over the past years, VSAs have been applied to a broad range of tasks and several VSA implementations have been proposed. The available implementations differ in the underlying vector space (e.g., binary vectors or complex-valued vectors) and the particular implementations of the required VSA operators - with important ramifications for the properties of these architectures. For example, not every VSA is equally well suited to address each task, including complete incompatibility. In this paper, we give an overview of eight available VSA implementations and discuss their commonalities and differences in the underlying vector space, bundling, and binding/unbinding operations. We create a taxonomy of available binding/unbinding operations and show an important ramification for non self-inverse binding operation using an example from analogical reasoning. A main contribution is the experimental comparison of the available implementations regarding (1) the capacity of bundles, (2) the approximation quality of non-exact unbinding operations, and (3) the influence of combined binding and bundling operations on the query answering performance. We expect this systematization and comparison to be relevant for development and evaluation of new VSAs, but most importantly, to support the selection of an appropriate VSA for a particular task.

[1]  Peter Protzel,et al.  A Neurologically Inspired Sequence Processing Model for Mobile Robot Place Recognition , 2019, IEEE Robotics and Automation Letters.

[2]  Jonathan Goldstein,et al.  When Is ''Nearest Neighbor'' Meaningful? , 1999, ICDT.

[3]  Dmitri A. Rachkovskij,et al.  SIMILARITY‐BASED RETRIEVAL WITH STRUCTURE‐SENSITIVE SPARSE BINARY DISTRIBUTED REPRESENTATIONS , 2012, Comput. Intell..

[4]  Stephen I. Gallant,et al.  Representing Objects, Relations, and Sequences , 2013, Neural Computation.

[5]  Aditya Joshi,et al.  Language Geometry Using Random Indexing , 2016, QI.

[6]  Paul Newman,et al.  1 year, 1000 km: The Oxford RobotCar dataset , 2017, Int. J. Robotics Res..

[7]  Geoffrey E. Hinton,et al.  Distributed representations and nested compositional structure , 1994 .

[8]  Richard Bellman,et al.  Adaptive Control Processes: A Guided Tour , 1961, The Mathematical Gazette.

[9]  Pentti Kanerva,et al.  Binary Spatter-Coding of Ordered K-Tuples , 1996, ICANN.

[10]  Paul Smolensky,et al.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1990, Artif. Intell..

[11]  Subutai Ahmad,et al.  Properties of Sparse Distributed Representations and their Application to Hierarchical Temporal Memory , 2015, ArXiv.

[12]  Richard Bellman,et al.  Adaptive Control Processes - A Guided Tour (Reprint from 1961) , 2015, Princeton Legacy Library.

[13]  T. Plate.  A Common Framework for Distributed Representation Schemes for Compositional Structure , 1997 .

[14]  Simon D. Levy,et al.  A distributed basis for analogical mapping , 2009 .

[15]  Gordon Wyeth,et al.  FAB-MAP + RatSLAM: Appearance-based SLAM for multiple times of day , 2010, 2010 IEEE International Conference on Robotics and Automation.

[16]  Denis Kleyko,et al.  Autoscaling Bloom filter: controlling trade-off between true and false positives , 2017, Neural Computing and Applications.

[17]  Luca Benini,et al.  Robust high-dimensional memory-augmented neural networks , 2020, Nature Communications.

[18]  S. Furber,et al.  To build a brain , 2012, IEEE Spectrum.

[19]  C. Pollard,et al.  Center for the Study of Language and Information , 2022 .

[20]  Magnus Sahlgren,et al.  Computing with large random patterns , 2001 .

[21]  Eric A. Weiss,et al.  The Hyperdimensional Stack Machine , 2018 .

[22]  Pentti Kanerva,et al.  Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors , 2009, Cognitive Computation.

[23]  Alexander Legalov,et al.  Associative synthesis of finite state automata model of a controlled object with hyperdimensional computing , 2017, IECON 2017 - 43rd Annual Conference of the IEEE Industrial Electronics Society.

[24]  Niko Sünderhauf,et al.  On the performance of ConvNet features for place recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[25]  Mark D. McDonnell,et al.  Enabling 'Question Answering' in the MBAT Vector Symbolic Architecture by Exploiting Orthogonal Random Matrices , 2014, 2014 IEEE International Conference on Semantic Computing.

[26]  Gordon Wyeth,et al.  SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights , 2012, 2012 IEEE International Conference on Robotics and Automation.

[27]  Tony A. Plate,et al.  Holographic Reduced Representation: Distributed Representation for Cognitive Structures , 2003 .

[28]  Pentti Kanerva,et al.  What We Mean When We Say "What's the Dollar of Mexico?": Prototypes and Mapping in Concept Space , 2010, AAAI Fall Symposium: Quantum Informatics for Cognitive, Social, and Semantic Processes.

[29]  Özgür Yilmaz,et al.  Symbolic Computation Using Cellular Automata-Based Hyperdimensional Computing , 2015, Neural Computation.

[30]  Chris Eliasmith,et al.  Vector-Derived Transformation Binding: An Improved Binding Operation for Deep Symbol-Like Processing in Neural Networks , 2019, Neural Computation.

[31]  Eamonn J. Keogh.  Nearest Neighbor , 2010, Encyclopedia of Machine Learning.

[32]  Ross W. Gayler.  Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience , 2004, ArXiv.

[33]  Friedrich T. Sommer,et al.  Variable Binding for Sparse Distributed Representations: Theory and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Bruno A. Olshausen,et al.  Superposition of many models into one , 2019, NeurIPS.

[35]  Takeo Kanade,et al.  Visual topometric localization , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[36]  Luca Benini,et al.  In-memory hyperdimensional computing , 2019, Nature Electronics.

[37]  Dmitri A. Rachkovskij,et al.  Representation and Processing of Structures with Binary Sparse Distributed Codes , 2001, IEEE Trans. Knowl. Data Eng..

[38]  Friedrich T. Sommer,et al.  A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks , 2018, Neural Computation.

[39]  Dominic Widdows,et al.  Geometry and Meaning , 2004, Computational Linguistics.

[40]  Dorothea Blostein,et al.  Encoding structure in holographic reduced representations. , 2013, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[41]  Peer Neubert,et al.  Hyperdimensional computing as a framework for systematic aggregation of image descriptors , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Niko Sünderhauf,et al.  Are We There Yet? Challenging SeqSLAM on a 3000 km Journey Across All Four Seasons , 2013 .

[43]  Nikolaos Papakonstantinou,et al.  Fault detection in the hyperspace: Towards intelligent automation systems , 2015, 2015 IEEE 13th International Conference on Industrial Informatics (INDIN).

[44]  Jan M. Rabaey,et al.  Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[46]  Wolfram Burgard,et al.  Probabilistic Robotics (Intelligent Robotics and Autonomous Agents) , 2005 .

[47]  Jan M. Rabaey,et al.  High-Dimensional Computing as a Nanoscalable Paradigm , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.

[48]  Peer Neubert,et al.  Vector Semantic Representations as Descriptors for Visual Place Recognition , 2021, Robotics: Science and Systems.

[49]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[50]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[51]  Subutai Ahmad,et al.  How Can We Be So Dense? The Benefits of Using Highly Sparse Representations , 2019, ArXiv.

[52]  Alex Graves,et al.  Associative Long Short-Term Memory , 2016, ICML.

[53]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[54]  Evgeny Osipov,et al.  Imitation of honey bees’ concept learning processes using Vector Symbolic Architectures , 2015, BICA 2015.

[55]  Denis Kleyko,et al.  Vector Symbolic Architectures and their applications: Computing with random vectors in a hyperdimensional space , 2018 .

[56]  Jussi H. Poikonen,et al.  High-dimensional computing with sparse vectors , 2015, 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[57]  Ross W. Gayler,et al.  Multiplicative Binding, Representation Operators & Analogy , 1998 .

[58]  Peer Neubert,et al.  Unsupervised Learning Methods for Visual Place Recognition in Discretely and Continuously Changing Environments , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[59]  Chris Eliasmith,et al.  How to Build a Brain: A Neural Architecture for Biological Cognition , 2013 .