Classification Using Hyperdimensional Computing: A Review

Hyperdimensional (HD) computing is built upon its unique data type referred to as hypervectors. The dimension of these hypervectors is typically in the range of tens of thousands. Proposed to solve cognitive tasks, HD computing aims at calculating similarity among its data. Data transformation is realized by three operations, including addition, multiplication and permutation. Its ultra-wide data representation introduces redundancy against noise. Since information is evenly distributed over every bit of the hypervectors, HD computing is inherently robust. Additionally, due to the nature of those three operations, HD computing leads to fast learning ability, high energy efficiency and acceptable accuracy in learning and classification tasks. This paper introduces the background of HD computing, and reviews the data representation, data transformation, and similarity measurement. The orthogonality in high dimensions presents opportunities for flexible computing. To balance the tradeoff between accuracy and efficiency, strategies include but are not limited to encoding, retraining, binarization and hardware acceleration. Evaluations indicate that HD computing shows great potential in addressing problems using data in the form of letters, signals and images. HD computing especially shows significant promise to replace machine learning algorithms as a light-weight classifier in the field of internet of things (IoTs).

[1]  Diederik Aerts,et al.  A comparison of geometric analogues of holographic reduced representations, original holographic reduced representations and binary spatter codes , 2011, 2011 Federated Conference on Computer Science and Information Systems (FedCSIS).

[2]  Beth Logan,et al.  Mel Frequency Cepstral Coefficients for Music Modeling , 2000, ISMIR.

[3]  Keshab K Parhi,et al.  Discriminative Ratio of Spectral Power and Relative Power Features Derived via Frequency-Domain Model Ratio With Application to Seizure Prediction , 2019, IEEE Transactions on Biomedical Circuits and Systems.

[4]  Okko Johannes Räsänen,et al.  Sequence Prediction With Sparse Distributed Hyperdimensional Coding Applied to the Analysis of Mobile Phone Use Patterns , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[5]  John Erik Hershey,et al.  A Hadamard Matrix , 1997 .

[6]  Mohsen Imani,et al.  VoiceHD: Hyperdimensional Computing for Efficient Speech Recognition , 2017, 2017 IEEE International Conference on Rebooting Computing (ICRC).

[7]  Özgür Yilmaz,et al.  Connectionist-Symbolic Machine Intelligence using Cellular Automata based Reservoir-Hyperdimensional Computing , 2015, ArXiv.

[8]  Y. Ahmet Sekercioglu,et al.  Holographic Graph Neuron: A Bioinspired Architecture for Pattern Processing , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Fan Wu,et al.  A Binary Learning Framework for Hyperdimensional Computing , 2019, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[10]  Michael N. Jones,et al.  Encoding Sequential Information in Semantic Space Models: Comparing Holographic Reduced Representation and Random Permutation , 2015, Comput. Intell. Neurosci..

[11]  Todorka Kovacheva,et al.  LINEAR CLASSIFIERS BASED ON BINARY DISTRIBUTED REPRESENTATIONS , 2007 .

[12]  Tajana Rosing,et al.  Hierarchical Hyperdimensional Computing for Energy Efficient Classification , 2018, 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC).

[13]  Jan M. Rabaey,et al.  Hyperdimensional Computing Nanosystem , 2018, ArXiv.

[14]  Nikolaos Papakonstantinou,et al.  Hyperdimensional Computing in Industrial Systems: The Use-Case of Distributed Fault Isolation in a Power Plant , 2018, IEEE Access.

[15]  Jan M. Rabaey,et al.  Classification and Recall With Binary Hyperdimensional Computing: Tradeoffs in Choice of Density and Mapping Characteristics , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[16]  Mohsen Imani,et al.  CompHD: Efficient Hyperdimensional Computing Using Model Compression , 2019, 2019 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED).

[17]  Jan M. Rabaey,et al.  Hyperdimensional Computing for Blind and One-Shot Classification of EEG Error-Related Potentials , 2020, Mob. Networks Appl..

[18]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[19]  Evgeny Osipov,et al.  Imitation of honey bees’ concept learning processes using Vector Symbolic Architectures , 2015, BICA 2015.

[20]  Stephen I. Gallant,et al.  Representing Objects, Relations, and Sequences , 2013, Neural Computation.

[21]  Mohsen Imani,et al.  QuantHD: A Quantization Framework for Hyperdimensional Computing , 2020, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[22]  Tajana Simunic,et al.  HDNA: Energy-efficient DNA sequencing using hyperdimensional computing , 2018, 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI).

[23]  Ross W. Gayler.  Multiplicative Binding, Representation Operators & Analogy (Workshop Poster) , 1998 .

[24]  Keshab K. Parhi,et al.  Seizure detection using regression tree based feature selection and polynomial SVM classification , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[25]  Evgeny Osipov,et al.  Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Evgeny Osipov,et al.  Modality classification of medical images with distributed representations based on cellular automata reservoir computing , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[27]  Luca Benini,et al.  Hardware Optimizations of Dense Binary Hyperdimensional Computing: Rematerialization of Hypervectors, Binarized Bundling, and Combinational Associative Memory , 2018, ACM J. Emerg. Technol. Comput. Syst..

[28]  Farinaz Koushanfar,et al.  SparseHD: Algorithm-Hardware Co-optimization for Efficient High-Dimensional Computing , 2019, 2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).

[29]  Aditya Joshi,et al.  Language Geometry Using Random Indexing , 2016, QI.

[30]  Evgeny Osipov,et al.  Integer Echo State Networks: Hyperdimensional Reservoir Computing , 2017, ArXiv.

[31]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[32]  Donald C. Wunsch,et al.  Application of random threshold neural networks for diagnostics of micro machine tool condition , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[33]  Francisco Herrera,et al.  Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study , 2015, Knowledge and Information Systems.

[34]  Anders Holst,et al.  Random indexing of text samples for latent semantic analysis , 2000 .

[35]  Alexander V. Terekhov,et al.  Hyper-dimensional computing for a visual question-answering system that is trainable end-to-end , 2017, ArXiv.

[36]  Ross W. Gayler.  Vector Symbolic Architectures answer Jackendoff's challenges for cognitive neuroscience , 2004, ArXiv.

[37]  Tony A. Plate,et al.  Holographic reduced representations , 1995, IEEE Trans. Neural Networks.

[38]  Keshab K. Parhi,et al.  Seizure detection using wavelet decomposition of the prediction error signal from a single channel of intra-cranial EEG , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[39]  E. M. Kussul,et al.  On image texture recognition by associative-projective neurocomputer , 1991 .

[40]  Keshab K. Parhi,et al.  MUSE: Minimum Uncertainty and Sample Elimination Based Binary Feature Selection , 2019, IEEE Transactions on Knowledge and Data Engineering.

[41]  Jan M. Rabaey,et al.  High-Dimensional Computing as a Nanoscalable Paradigm , 2017, IEEE Transactions on Circuits and Systems I: Regular Papers.

[42]  Okko Johannes Räsänen,et al.  Modeling Dependencies in Multiple Parallel Data Streams with Hyperdimensional Computing , 2014, IEEE Signal Processing Letters.

[43]  Giovanni De Micheli,et al.  AdaptHD: Adaptive Efficient Training for Brain-Inspired Hyperdimensional Computing , 2019, 2019 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[44]  Luca Benini,et al.  In-memory hyperdimensional computing , 2019, Nature Electronics.

[45]  Jan M. Rabaey,et al.  Hyperdimensional Computing Exploiting Carbon Nanotube FETs, Resistive RAM, and Their Monolithic 3D Integration , 2018, IEEE Journal of Solid-State Circuits.

[46]  Pentti Kanerva,et al.  Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High-Dimensional Random Vectors , 2009, Cognitive Computation.

[47]  Luca Benini,et al.  Laelaps: An Energy-Efficient Seizure Detection Algorithm from Long-term Human iEEG Recordings without False Alarms , 2019, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[48]  Okko Johannes Räsänen,et al.  Generating Hyperdimensional Distributed Representations from Continuous-Valued Multivariate Sensory Input , 2015, CogSci.

[49]  Jan M. Rabaey,et al.  A Robust and Energy-Efficient Classifier Using Brain-Inspired Hyperdimensional Computing , 2016, ISLPED.

[50]  Luca Benini,et al.  One-shot Learning for iEEG Seizure Detection Using End-to-end Binary Operations: Local Binary Patterns with Hyperdimensional Computing , 2018, 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[51]  Evgeny Osipov,et al.  Brain-like classifier of temporal patterns , 2014, 2014 International Conference on Computer and Information Sciences (ICCOINS).

[52]  Alexander G. Anderson,et al.  The High-Dimensional Geometry of Binary Neural Networks , 2017, ICLR.

[53]  Alec Xavier Manabat,et al.  Performance Analysis of Hyperdimensional Computing for Character Recognition , 2019, 2019 International Symposium on Multimedia and Communication Technology (ISMAC).

[54]  Luca Benini,et al.  Efficient Biosignal Processing Using Hyperdimensional Computing: Network Templates for Combined Learning and Classification of ExG Signals , 2019, Proceedings of the IEEE.

[55]  Peer Neubert,et al.  A comparison of vector symbolic architectures , 2020, Artificial Intelligence Review.

[56]  Luca Benini,et al.  Hyperdimensional biosignal processing: A case study for EMG-based hand gesture recognition , 2016, 2016 IEEE International Conference on Rebooting Computing (ICRC).

[57]  Luca Benini,et al.  Evolvable Hyperdimensional Computing: Unsupervised Regeneration of Associative Memory to Recover Faulty Components , 2020, 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS).

[58]  Daswin De Silva,et al.  Trajectory clustering of road traffic in urban environments using incremental machine learning in combination with hyperdimensional computing , 2019, 2019 IEEE Intelligent Transportation Systems Conference (ITSC).

[59]  Dmitri A. Rachkovskij,et al.  Binding and Normalization of Binary Sparse Distributed Representations by Context-Dependent Thinning , 2001, Neural Computation.

[60]  Marcus Liwicki,et al.  HyperEmbed: Tradeoffs Between Resources and Performance in NLP Tasks with Hyperdimensional Computing enabled Embedding of n-gram Statistics , 2020, ArXiv.

[61]  Keshab K. Parhi,et al.  Seizure prediction using polynomial SVM classification , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[62]  Jan M. Rabaey,et al.  A Programmable Hyper-Dimensional Processor Architecture for Human-Centric IoT , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[63]  Yiannis Aloimonos,et al.  Learning sensorimotor control with neuromorphic sensors: Toward hyperdimensional active perception , 2019, Science Robotics.

[64]  Jan M. Rabaey,et al.  Hyperdimensional computing with 3D VRRAM in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition , 2016, 2016 IEEE International Electron Devices Meeting (IEDM).

[65]  David R. O'Hallaron,et al.  Computer Systems: A Programmer's Perspective , 1991 .

[66]  R. Cuninghame-Green,et al.  Applied Linear Algebra , 1979 .

[67]  Giovanni De Micheli,et al.  QubitHD: A Stochastic Acceleration Method for HD Computing-Based Machine Learning , 2019, ArXiv.

[68]  Tajana Simunic,et al.  A Framework for Collaborative Learning in Secure High-Dimensional Space , 2019, 2019 IEEE 12th International Conference on Cloud Computing (CLOUD).

[69]  Jan M. Rabaey,et al.  Hyperdimensional computing for noninvasive brain-computer interfaces: Blind and one-shot classification of EEG error-related potentials , 2017 .

[70]  Pentti Kanerva,et al.  Fully Distributed Representation , 1997 .

[71]  Geoffrey E. Hinton.  Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems , 1991 .

[72]  Luca Benini,et al.  An EMG Gesture Recognition System with Flexible High-Density Sensors and Brain-Inspired High-Dimensional Classifier , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[73]  Dmitri A. Rachkovskij,et al.  Representation and Processing of Structures with Binary Sparse Distributed Codes , 2001, IEEE Trans. Knowl. Data Eng..

[74]  Luca Benini,et al.  Exploring Embedding Methods in Binary Hyperdimensional Computing: A Case Study for Motor-Imagery based Brain-Computer Interfaces , 2018, ArXiv.

[75]  Keshab K. Parhi,et al.  Low-Complexity Seizure Prediction From iEEG/sEEG Using Spectral Power and Ratios of Spectral Power , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[76]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[77]  P. Kanerva,et al.  Hyperdimensional Computing for Text Classification , 2016 .

[78]  Valeriy Vyatkin,et al.  Distributed Representation of n-gram Statistics for Boosting Self-organizing Maps with Hyperdimensional Computing , 2019, Ershov Informatics Conference.

[79]  Tajana Simunic,et al.  SemiHD: Semi-Supervised Learning Using Hyperdimensional Computing , 2019, 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[80]  Tajana Simunic,et al.  F5-HD: Fast Flexible FPGA-based Framework for Refreshing Hyperdimensional Computing , 2019, FPGA.