A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations

This two-part comprehensive survey is devoted to a computing framework most commonly known under the names Hyperdimensional Computing and Vector Symbolic Architectures (HDC/VSA). Both names refer to a family of computational models that use highdimensional distributed representations and rely on the algebraic properties of their key operations to incorporate the advantages of structured symbolic representations and vector distributed representations. Notable models in the HDC/VSA family are Tensor Product Representations, Holographic Reduced Representations, Multiply-Add-Permute, Binary Spatter Codes, and Sparse Binary Distributed Representations but there are other models too. HDC/VSA is a highly interdisciplinary area with connections to computer science, electrical engineering, artificial intelligence, mathematics, and cognitive science. This fact makes it challenging to create a thorough overview of the area. However, due to a surge of new researchers joining the area in recent years, the necessity for a comprehensive survey of the area has become extremely important. Therefore, amongst other aspects of the area, this Part I surveys important aspects such as: known computational models of HDC/VSA and transformations of various input data types to high-dimensional distributed representations. Part II of this survey [Kleyko et al., 2021c] is devoted to applications, cognitive computing and architectures, as well as directions for future work. The survey is written to be useful for both newcomers and practitioners.

[1]  Dmitri A. Rachkovskij,et al.  Building a world model with structure-sensitive sparse binary distributed representations , 2013, BICA 2013.

[2]  Douglas Summers-Stay,et al.  Symbolic Representation and Learning With Hyperdimensional Computing , 2020, Frontiers in Robotics and AI.

[3]  Alex Graves,et al.  Associative Long Short-Term Memory , 2016, ICML.

[4]  Agnieszka Patyk,et al.  Geometric Algebra Model of Distributed Representations , 2010, Geometric Algebra Computing.

[5]  Trevor Cohen,et al.  Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (EARP) , 2018, CoNLL.

[6]  Bart De Moor,et al.  Geometric Analogue of Holographic Reduced Representation , 2007, ArXiv.

[7]  Ernst Kussul,et al.  Improved Method of Handwritten Digit Recognition , 2002 .

[8]  Samuel Kaski,et al.  Dimensionality reduction by random mapping: fast similarity computation for clustering , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[9]  Diederik Aerts,et al.  Tensor-product versus geometric-product coding , 2007, 0709.1268.

[10]  Pentti Kanerva,et al.  Binary Spatter-Coding of Ordered K-Tuples , 1996, ICANN.

[11]  Donald C. Wunsch,et al.  The random subspace coarse coding scheme for real-valued vectors , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[12]  D. Smith,et al.  A random walk in Hamming space , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  Özgür Yilmaz,et al.  Symbolic Computation Using Cellular Automata-Based Hyperdimensional Computing , 2015, Neural Computation.

[14]  James S. Albus,et al.  Data Storage in the Cerebellar Model Articulation Controller (CMAC) , 1975 .

[15]  Diederik Aerts,et al.  Distributed Representations Based on Geometric Algebra: the Continuous Model , 2011, Informatica.

[16]  Magnus Sahlgren,et al.  An Introduction to Random Indexing , 2005 .

[17]  Phil Culliton,et al.  Positional binding with distributed representations , 2016, 2016 International Conference on Image, Vision and Computing (ICIVC).

[18]  Trevor Cohen,et al.  Orthogonality and Orthography: Introducing Measured Distance into Semantic Space , 2013, QI.

[19]  Jan M. Rabaey,et al.  Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[20]  R. Jackendoff.  Foundations of Language: Brain, Meaning, Grammar, Evolution , 2002 .

[21]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[22]  Pentti Kanerva.  Computing with High-Dimensional Vectors , 2019, IEEE Design & Test.

[23]  Luca Benini,et al.  Hyperdimensional biosignal processing: A case study for EMG-based hand gesture recognition , 2016, 2016 IEEE International Conference on Rebooting Computing (ICRC).

[24]  Chris Eliasmith,et al.  Vector-Derived Transformation Binding: An Improved Binding Operation for Deep Symbol-Like Processing in Neural Networks , 2019, Neural Computation.

[25]  Ross W. Gayler.  Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience , 2004, ArXiv.

[26]  Friedrich T. Sommer,et al.  Variable Binding for Sparse Distributed Representations: Theory and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[27]  John E. Hummel,et al.  Distributed representations of structure: A theory of analogical access and mapping. , 1997 .

[28]  Pentti Kanerva.  Computing with 10,000-bit words , 2014, 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[29]  Petri Kanerva,et al.  Dual role of analogy in the design of a cognitive computer , 1998 .

[30]  Donald C. Wunsch,et al.  Neural Networks and Micromechanics , 2009 .

[31]  Sanjay J. Patel,et al.  Characterizing the effects of transient faults on a high-performance processor pipeline , 2004, International Conference on Dependable Systems and Networks, 2004.

[32]  Jan M. Rabaey,et al.  High-Dimensional Computing as a Nanoscalable Paradigm , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  S. V. Slipchenko,et al.  Randomized projective methods for the construction of binary sparse vector representations , 2012, Cybernetics and Systems Analysis.

[34]  Peer Neubert,et al.  Towards Hypervector Representations for Learning and Planning with Schemas , 2018, KI.

[35]  Paul Thagard,et al.  Concepts as Semantic Pointers: A Framework and Computational Model , 2016, Cogn. Sci..

[36]  Terrence C. Stewart,et al.  A biologically realistic cleanup memory: Autoassociation in spiking neurons , 2011, Cognitive Systems Research.

[37]  Dmitri A. Rachkovskij,et al.  A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges , 2021, ArXiv.

[38]  Dmitri A. Rachkovskij,et al.  Representation and Processing of Structures with Binary Sparse Distributed Codes , 2001, IEEE Trans. Knowl. Data Eng..

[39]  Dimitris Achlioptas,et al.  Database-friendly random projections: Johnson-Lindenstrauss with binary coins , 2003, J. Comput. Syst. Sci..

[40]  Jan M. Rabaey,et al.  Vector Symbolic Architectures as a Computing Framework for Nanoscale Hardware , 2021, ArXiv.

[41]  Eric A. Weiss,et al.  The Hyperdimensional Stack Machine , 2018 .

[42]  Distributed vs. Local Representation , 2022 .

[43]  Kenneth Ward Church,et al.  Very sparse random projections , 2006, KDD '06.

[44]  Michael N Jones,et al.  Representing word meaning and order information in a composite holographic lexicon. , 2007, Psychological review.

[45]  Mark D. McDonnell,et al.  Enabling 'Question Answering' in the MBAT Vector Symbolic Architecture by Exploiting Orthogonal Random Matrices , 2014, 2014 IEEE International Conference on Semantic Computing.

[46]  Friedrich T. Sommer,et al.  A Theory of Sequence Indexing and Working Memory in Recurrent Neural Networks , 2018, Neural Computation.

[47]  Yiannis Aloimonos,et al.  Learning sensorimotor control with neuromorphic sensors: Toward hyperdimensional active perception , 2019, Science Robotics.

[48]  Dmitri A. Rachkovskij,et al.  Neural Distributed Autoassociative Memories: A Survey , 2017, ArXiv.

[49]  Ernst Kussul,et al.  Sparse Binary Distributed Encoding of Numeric Vectors , 2005 .

[50]  Burton H. Bloom,et al.  Space/time trade-offs in hash coding with allowable errors , 1970, CACM.

[51]  Javier Snaider,et al.  Modular Composite Representation , 2014, Cognitive Computation.

[52]  W. B. Johnson,et al.  Extensions of Lipschitz mappings into Hilbert space , 1984 .

[53]  Pentti Kanerva,et al.  The Spatter Code for Encoding Concepts at Many Levels , 1994 .

[54]  D. Rachkovskij.  Formation of Similarity-Reflecting Binary Vectors with Random Binary Projections , 2015, Cybernetics and Systems Analysis.

[55]  Tajana Simunic,et al.  HDNA: Energy-efficient DNA sequencing using hyperdimensional computing , 2018, 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[56]  Jan M. Rabaey,et al.  Hyperdimensional Computing Exploiting Carbon Nanotube FETs, Resistive RAM, and Their Monolithic 3D Integration , 2018, IEEE Journal of Solid-State Circuits.

[57]  Bruno A. Olshausen,et al.  Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures , 2020, Neural Computation.

[58]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[59]  Brent Komer,et al.  Efficient navigation using a scalable, biologically inspired spatial representation , 2020, CogSci.

[60]  Luca Benini,et al.  Near-channel classifier: symbiotic communication and classification in high-dimensional space , 2021, Brain Informatics.

[61]  Michael N. Jones,et al.  Encoding Sequential Information in Semantic Space Models: Comparing Holographic Reduced Representation and Random Permutation , 2015, Comput. Intell. Neurosci..

[62]  Pentti Kanerva,et al.  Fully Distributed Representation , 1997 .

[63]  Agnieszka Patyk-Lonska,et al.  Preserivng pieces of information in a given order in HRR and GAc , 2011, 2011 Federated Conference on Computer Science and Information Systems (FedCSIS).

[64]  Dmitri A. Rachkovskij,et al.  Modification of Holographic Graph Neuron Using Sparse Distributed Representations , 2016 .

[65]  Özgür Yilmaz,et al.  Machine Learning Using Cellular Automata Based Feature Expansion and Reservoir Computing , 2015, J. Cell. Autom..

[66]  Anders Holst,et al.  Random indexing of text samples for latent semantic analysis , 2000 .

[67]  Anja Becker,et al.  New directions in nearest neighbor searching with applications to lattice sieving , 2016, IACR Cryptol. ePrint Arch..

[68]  Aditya Joshi,et al.  Language Geometry Using Random Indexing , 2016, QI.

[69]  Evgeny Osipov,et al.  Recognizing Permuted Words with Vector Symbolic Architectures: A Cambridge Test for Machines , 2016, BICA.

[70]  Michael N. Jones,et al.  OrBEAGLE: Integrating Orthography into a Holographic Model of the Lexicon , 2011, ICANN.

[71]  Magnus Sahlgren,et al.  Encoding Sequential Information in Vector Space Models of Semantics: Comparing Holographic Reduced Representation and Random Permutation , 2010 .

[72]  Peer Neubert,et al.  An Introduction to Hyperdimensional Computing for Robotics , 2019, KI - Künstliche Intelligenz.

[73]  Zenon W. Pylyshyn,et al.  Connectionism and cognitive architecture: A critical analysis , 1988, Cognition.

[74]  Volker Tresp,et al.  Holistic Representations for Memorization and Inference , 2018, UAI.

[75]  P. Kanerva,et al.  Permutations as a means to encode order in word space , 2008 .

[76]  C. Koch,et al.  Invariant visual representation by single neurons in the human brain , 2005, Nature.

[77]  Tajana Simunic,et al.  GenieHD: Efficient DNA Pattern Matching Accelerator Using Hyperdimensional Computing , 2020, 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[78]  M. Aizerman,et al.  Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning , 1964 .

[79]  Stephen I. Gallant,et al.  Representing Objects, Relations, and Sequences , 2013, Neural Computation.

[80]  Tony A. Plate,et al.  Analogy retrieval and processing with distributed vector representations , 2000, Expert Syst. J. Knowl. Eng..

[81]  Ivan Tyukin,et al.  Blessing of dimensionality: mathematical foundations of the statistical physics of data , 2018, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[82]  Daniel M. Kane,et al.  Sparser Johnson-Lindenstrauss Transforms , 2010, JACM.

[83]  E. Mizraji.  Vector logics: the matrix-vector representation of logical calculus , 1992 .

[84]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[85]  T. Plate.  A Common Framework for Distributed Representation Schemes for Compositional Structure , 1997 .

[86]  S. V. Slipchenko,et al.  Resolution of Binary Coding of Real-Valued Vectors by Hyperrectangular Receiptive Fields , 2005 .

[87]  Emmanuel Dupoux,et al.  Holographic String Encoding , 2011, Cogn. Sci..

[88]  L. Benini,et al.  A 5 μW Standard Cell Memory-Based Configurable Hyperdimensional Computing Accelerator for Always-on Smart Sensing , 2021, IEEE Transactions on Circuits and Systems I: Regular Papers.

[89]  Douglas Summers-Stay,et al.  Representing Sets as Summed Semantic Vectors , 2018, Biologically Inspired Cognitive Architectures.

[90]  Trevor Bekolay,et al.  A Large-Scale Model of the Functioning Brain , 2012, Science.

[91]  Friedrich T. Sommer,et al.  Robust computation with rhythmic spike patterns , 2019, Proceedings of the National Academy of Sciences.

[92]  Luca Benini,et al.  Efficient Biosignal Processing Using Hyperdimensional Computing: Network Templates for Combined Learning and Classification of ExG Signals , 2019, Proceedings of the IEEE.

[93]  Geoffrey E. Hinton,et al.  Distributed representations and nested compositional structure , 1994 .

[94]  Santosh S. Vempala,et al.  Latent semantic indexing: a probabilistic analysis , 1998, PODS '98.

[95]  Luca Benini,et al.  In-memory hyperdimensional computing , 2019, Nature Electronics.

[96]  Andrew McCallum,et al.  Energy and Policy Considerations for Deep Learning in NLP , 2019, ACL.

[97]  Hun-Seok Kim,et al.  HDM: Hyper-Dimensional Modulation for Robust Low-Power Communications , 2018, 2018 IEEE International Conference on Communications (ICC).

[98]  Ernst Kussul,et al.  Permutative coding technique for handwritten digit recognition system , 2003, Proceedings of the International Joint Conference on Neural Networks, 2003..

[99]  Luca Benini,et al.  Hardware Optimizations of Dense Binary Hyperdimensional Computing: Rematerialization of Hypervectors, Binarized Bundling, and Combinational Associative Memory , 2018, ACM J. Emerg. Technol. Comput. Syst..

[100]  Terrence C. Stewart,et al.  A neural representation of continuous space using fractional binding , 2019, CogSci.

[101]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[102]  Bruno A. Olshausen,et al.  Resonator Networks, 2: Factorization Performance and Capacity Compared to Optimization-Based Methods , 2020, Neural Computation.

[103]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[104]  Jan M. Rabaey,et al.  Hyperdimensional computing with 3D VRRAM in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition , 2016, 2016 IEEE International Electron Devices Meeting (IEDM).

[105]  A. Sokolov,et al.  APPROACHES TO SEQUENCE SIMILARITY REPRESENTATION , 2006 .

[106]  Bruno A. Olshausen,et al.  Computing on Functions Using Randomized Vector Representations (in brief) , 2021, NICE.

[107]  Richard W. Prager.  Networks based on Kanerva's sparse distributed memory: results showing their strengths and limitations and a new algorithm to design the location matching layer , 1993, IEEE International Conference on Neural Networks.

[108]  Pentti Kanerva,et al.  Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors , 2009, Cognitive Computation.

[109]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[110]  P. Milner.  A model for visual shape recognition. , 1974, Psychological review.

[111]  Simon D. Levy,et al.  A distributed basis for analogical mapping , 2009 .

[112]  Denis Kleyko,et al.  Autoscaling Bloom filter: controlling trade-off between true and false positives , 2017, Neural Computing and Applications.

[113]  Geoffrey E. Hinton,et al.  Implementing Semantic Networks in Parallel Hardware , 2014 .

[114]  Tony Plate,et al.  Holographic Reduced Representations: Convolution Algebra for Compositional Distributed Representations , 1991, IJCAI.

[116]  S. Thorpe.  Localized versus distributed representations , 1998 .

[117]  E Mizraji,et al.  Context-dependent associations in linear distributed memories. , 1989, Bulletin of mathematical biology.

[118]  Ernst Kussul,et al.  Properties of Numeric Codes for the Scheme of Random Subspaces RSC , 2005 .

[119]  Diederik Aerts,et al.  A comparison of geometric analogues of holographic reduced representations, original holographic reduced representations and binary spatter codes , 2011, 2011 Federated Conference on Computer Science and Information Systems (FedCSIS).

[120]  Luca Benini,et al.  Robust high-dimensional memory-augmented neural networks , 2020, Nature Communications.

[121]  Jörg Conradt,et al.  Analyzing the Capacity of Distributed Vector Representations to Encode Spatial Information , 2020, 2020 International Joint Conference on Neural Networks (IJCNN).

[122]  E. M. Kussul,et al.  On image texture recognition by associative-projective neurocomputer , 1991 .

[123]  Y. Ahmet Sekercioglu,et al.  Holographic Graph Neuron: A Bioinspired Architecture for Pattern Processing , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[124]  Lorenzo Rosasco,et al.  Holographic Embeddings of Knowledge Graphs , 2015, AAAI.

[125]  Dorothea Blostein,et al.  Encoding structure in holographic reduced representations. , 2013, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[126]  Friedrich T. Sommer,et al.  Cellular Automata Can Reduce Memory Requirements of Collective-State Computing , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[127]  D. Rachkovskij,et al.  Vector Data Transformation Using Random Binary Matrices , 2014, Cybernetics and Systems Analysis.

[128]  C. von der Malsburg,et al.  Am I Thinking Assemblies , 1986 .

[129]  Terrence C. Stewart,et al.  Sentence processing in spiking neurons: A biologically plausible left-corner parser , 2014, CogSci.

[130]  L. Shastri,et al.  From simple associations to systematic reasoning: A connectionist representation of rules, variables and dynamic bindings using temporal synchrony , 1993, Behavioral and Brain Sciences.

[131]  Sanjoy Dasgupta,et al.  A neural algorithm for a fundamental computing problem , 2017 .

[132]  Agnieszka Patyk-Lonska,et al.  Experiments on Preserving Pieces of Information in a Given Order in Holographic Reduced Representations and the Continuous Geometric Algebra Model , 2011, Informatica.

[133]  Simon D. Levy,et al.  Vector Symbolic Architectures: A New Building Material for Artificial General Intelligence , 2008, AGI.

[134]  Okko Johannes Räsänen,et al.  Generating Hyperdimensional Distributed Representations from Continuous-Valued Multivariate Sensory Input , 2015, CogSci.

[135]  Jussi H. Poikonen,et al.  High-dimensional computing with sparse vectors , 2015, 2015 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[136]  Keshab K. Parhi,et al.  Classification Using Hyperdimensional Computing: A Review , 2020, IEEE Circuits and Systems Magazine.

[137]  Geoffrey E. Hinton.  Mapping Part-Whole Hierarchies into Connectionist Networks , 1990, Artif. Intell..

[138]  Ross W. Gayler,et al.  Multiplicative Binding, Representation Operators & Analogy , 1998 .

[139]  Tatiana Baidyk,et al.  Improved method of handwritten digit recognition tested on MNIST database , 2004, Image Vis. Comput..

[140]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[141]  Denis Kleyko,et al.  Commentaries on "Learning Sensorimotor Control with Neuromorphic Sensors: Toward Hyperdimensional Active Perception" [Science Robotics Vol 4 Issue 30 (2019) 1-10 , 2020, ArXiv.

[142]  Alec Xavier Manabat,et al.  Performance Analysis of Hyperdimensional Computing for Character Recognition , 2019, 2019 International Symposium on Multimedia and Communication Technology (ISMAC).

[143]  Tony Plate,et al.  Holographic Recurrent Networks , 1992, NIPS.

[144]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[145]  Baker Mohammad,et al.  Hyper-Dimensional Computing Challenges and Opportunities for AI Applications , 2021, IEEE Access.