Biologically-Inspired Pulse Signal Processing for Intelligence at the Edge

There is an ever-growing mismatch between the proliferation of data-intensive, power-hungry deep learning solutions in the machine learning (ML) community and the need for agile, portable solutions in resource-constrained devices, particularly for intelligence at the edge. In this paper, we present a fundamentally novel approach that leverages data-driven intelligence with biologically-inspired efficiency. The proposed Sparse Embodiment Neural-Statistical Architecture (SENSA) decomposes the learning task into two distinct phases: a training phase and a hardware embedment phase where prototypes are extracted from the trained network and used to construct fast, sparse embodiment for hardware deployment at the edge. Specifically, we propose the Sparse Pulse Automata via Reproducing Kernel (SPARK) method, which first constructs a learning machine in the form of a dynamical system using energy-efficient spike or pulse trains, commonly used in neuroscience and neuromorphic engineering, then extracts a rule-based solution in the form of automata or lookup tables for rapid deployment in edge computing platforms. We propose to use the theoretically-grounded unifying framework of the Reproducing Kernel Hilbert Space (RKHS) to provide interpretable, nonlinear, and nonparametric solutions, compared to the typical neural network approach. In kernel methods, the explicit representation of the data is of secondary nature, allowing the same algorithm to be used for different data types without altering the learning rules. To showcase SPARK’s capabilities, we carried out the first proof-of-concept demonstration on the task of isolated-word automatic speech recognition (ASR) or keyword spotting, benchmarked on the TI-46 digit corpus. Together, these energy-efficient and resource-conscious techniques will bring advanced machine learning solutions closer to the edge.

[1]  Tara N. Sainath,et al.  Kernel methods match Deep Neural Networks on TIMIT , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[2]  Liam McDaid,et al.  SWAT: A Spiking Neural Network Training Algorithm for Classification Problems , 2010, IEEE Transactions on Neural Networks.

[3]  Jason Weston,et al.  A unified architecture for natural language processing: deep neural networks with multitask learning , 2008, ICML '08.

[4]  Weifeng Liu,et al.  Kernel Adaptive Filtering: A Comprehensive Introduction , 2010 .

[5]  José Carlos Príncipe,et al.  The integrate-and-fire sampler: A special type of asynchronous Σ - Δ modulator , 2011, 2011 IEEE International Symposium of Circuits and Systems (ISCAS).

[6]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[7]  Benjamin Schrauwen,et al.  Recognition of Isolated Digits using a Liquid State Machine , 2005 .

[8]  M. Picheny,et al.  Comparison of Parametric Representation for Monosyllabic Word Recognition in Continuously Spoken Sentences , 2017 .

[9]  Steve B. Furber,et al.  The SpiNNaker Project , 2014, Proceedings of the IEEE.

[10]  Noam Chomsky,et al.  Three models for the description of language , 1956, IRE Trans. Inf. Theory.

[11]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[12]  Robert A. Legenstein,et al.  Methods for Estimating the Computational Power and Generalization Capability of Neural Microcircuits , 2004, NIPS.

[13]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[14]  E. Mark Gold,et al.  Complexity of Automaton Identification from Given Data , 1978, Inf. Control..

[15]  Alaa A. Kharbouch,et al.  Three models for the description of language , 1956, IRE Trans. Inf. Theory.

[16]  Carl H. Smith,et al.  Inductive Inference: Theory and Methods , 1983, CSUR.

[17]  Andrew S. Cassidy,et al.  A million spiking-neuron integrated circuit with a scalable communication network and interface , 2014, Science.

[18]  N. Aronszajn Theory of Reproducing Kernels. , 1950 .

[19]  Stan Davis,et al.  Comparison of Parametric Representations for Monosyllabic Word Recognition in Continuously Spoken Se , 1980 .

[20]  Andrew Gordon Wilson,et al.  Stochastic Variational Deep Kernel Learning , 2016, NIPS.

[21]  José Carlos Príncipe,et al.  A Reproducing Kernel Hilbert Space Framework for Spike Train Signal Processing , 2009, Neural Computation.

[22]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[23]  José Carlos Príncipe,et al.  Approximate reconstruction of bandlimited functions for the integrate and fire sampler , 2009, Advances in Computational Mathematics.

[24]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[25]  David H. Wolpert,et al.  The Lack of A Priori Distinctions Between Learning Algorithms , 1996, Neural Computation.

[26]  Kan Li,et al.  Surprise-Novelty Information Processing for Gaussian Online Active Learning (SNIP-GOAL) , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[27]  Weifeng Liu,et al.  Kernel Adaptive Filtering , 2010 .

[28]  Kan Li,et al.  Transfer Learning in Adaptive Filters: The Nearest Instance Centroid-Estimation Kernel Least-Mean-Square Algorithm , 2017, IEEE Transactions on Signal Processing.

[29]  Shouyi Yin,et al.  An Ultra-Low Power Binarized Convolutional Neural Network-Based Speech Recognition Processor With On-Chip Self-Learning , 2019, IEEE Transactions on Circuits and Systems I: Regular Papers.

[30]  Weisong Shi,et al.  Edge Computing: Vision and Challenges , 2016, IEEE Internet of Things Journal.

[31]  José Carlos Príncipe,et al.  Strictly Positive-Definite Spike Train Kernels for Point-Process Divergences , 2012, Neural Computation.

[32]  José Carlos Príncipe,et al.  Theory and Algorithms for Pulse Signal Processing , 2018, IEEE Transactions on Circuits and Systems I: Regular Papers.

[33]  Lawrence K. Saul,et al.  Kernel Methods for Deep Learning , 2009, NIPS.

[34]  Kan Li,et al.  Restoring Behavior via Inverse Neurocontroller in a Lesioned Cortical Spiking Model Driving a Virtual Arm , 2016, Front. Neurosci..

[35]  Shih-Chii Liu,et al.  Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification , 2017, Front. Neurosci..

[36]  Wolfgang Maass,et al.  Networks of Spiking Neurons: The Third Generation of Neural Network Models , 1996, Electron. Colloquium Comput. Complex..

[37]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[38]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[39]  Yong Zhang,et al.  A Digital Liquid State Machine With Biologically Inspired Learning and Its Application to Speech Recognition , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[40]  Kan Li,et al.  Biologically-Inspired Spike-Based Automatic Speech Recognition of Isolated Digits Over a Reproducing Kernel Hilbert Space , 2018, Front. Neurosci..

[41]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[42]  José Carlos Príncipe,et al.  Kernel Methods on Spike Train Space for Neuroscience: A Tutorial , 2013, IEEE Signal Processing Magazine.

[43]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[44]  Kan Li,et al.  The Kernel Adaptive Autoregressive-Moving-Average Algorithm , 2016, IEEE Transactions on Neural Networks and Learning Systems.