High-Density Liquid-State Machine Circuitry for Time-Series Forecasting

Spiking neural networks (SNN) are the last neural network generation that try to mimic the real behavior of biological neurons. Although most research in this area is done through software applications, it is in hardware implementations in which the intrinsic parallelism of these computing systems are more efficiently exploited. Liquid state machines (LSM) have arisen as a strategic technique to implement recurrent designs of SNN with a simple learning methodology. In this work, we show a new low-cost methodology to implement high-density LSM by using Boolean gates. The proposed method is based on the use of probabilistic computing concepts to reduce hardware requirements, thus considerably increasing the neuron count per chip. The result is a highly functional system that is applied to high-speed time series forecasting.

[1]  Grzegorz M. Wójcik,et al.  Liquid State Machine Built of Hodgkin-Huxley Neurons , 2004, Informatica.

[2]  Stefan Schliebs,et al.  Are probabilistic spiking neural networks suitable for reservoir computing? , 2011, The 2011 International Joint Conference on Neural Networks.

[3]  S. Bressler.  Large-scale cortical networks and cognition , 1995, Brain Research Reviews.

[4]  Enrique Fernández-Blanco,et al.  Artificial Neuron-Glia Networks Learning Approach Based on Cooperative Coevolution , 2015, Int. J. Neural Syst..

[5]  John D. Enderle,et al.  A New Linear muscle Fiber Model for Neural Control of saccades , 2013, Int. J. Neural Syst..

[6]  Kishan G. Mehrotra,et al.  Forecasting the behavior of multivariate time series using neural networks , 1992, Neural Networks.

[7]  Simei Gomes Wysoski,et al.  Fast and adaptive network of spiking neurons for multi-view visual pattern recognition , 2008, Neurocomputing.

[8]  Yong Zhang,et al.  A Digital Liquid State Machine With Biologically Inspired Learning and Its Application to Speech Recognition , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Antoni Morro,et al.  Studying the Role of Synchronized and Chaotic Spiking Neural Ensembles in Neural Information Processing , 2014, Int. J. Neural Syst..

[10]  Benjamin Schrauwen,et al.  Compact hardware for real-time speech recognition using a Liquid State Machine , 2007, 2007 International Joint Conference on Neural Networks.

[11]  Jiwen Dong,et al.  Time-series forecasting using flexible neural tree model , 2005, Inf. Sci..

[12]  Alireza Ghahari,et al.  A Neuron-Based Time-Optimal controller of Horizontal Saccadic eye movements , 2014, Int. J. Neural Syst..

[13]  Christof Koch,et al.  Subthreshold Voltage Noise Due to Channel Fluctuations in Active Neuronal Membranes , 2000, Journal of Computational Neuroscience.

[14]  Walter Senn,et al.  Code-Specific Learning Rules Improve Action Selection by Populations of Spiking Neurons , 2014, Int. J. Neural Syst..

[15]  M. London,et al.  Sensitivity to perturbations in vivo implies high noise and suggests rate coding in cortex , 2010, Nature.

[16]  Xiaosi Zeng,et al.  Development of Recurrent Neural Network Considering Temporal‐Spatial Input Dynamics for Freeway Travel Time Modeling , 2013, Comput. Aided Civ. Infrastructure Eng..

[17]  Benjamin Schrauwen,et al.  Compact hardware liquid state machines on FPGA for real-time speech recognition , 2008, Neural Networks.

[18]  Vicent Canals,et al.  Self-configuring spiking neural networks , 2008, IEICE Electron. Express.

[19]  Hojjat Adeli,et al.  Parallel backpropagation learning algorithms on CRAY Y-MP8/864 supercomputer , 1993, Neurocomputing.

[20]  E. Ivo Alves,et al.  Earthquake Forecasting Using Neural Networks: Results and Future Work , 2006 .

[21]  Marcus O'Connor,et al.  Artificial neural network models for forecasting and decision making , 1994 .

[22]  Snjezana Soltic,et al.  Knowledge Extraction from Evolving Spiking Neural Networks with Rank Order Population Coding , 2010, Int. J. Neural Syst..

[23]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[24]  Benjamin Schrauwen,et al.  Toward optical signal processing using photonic reservoir computing. , 2008, Optics express.

[25]  Teresa Bernarda Ludermir,et al.  Investigating the use of Reservoir Computing for forecasting the hourly wind speed in short -term , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[26]  Ian Flood,et al.  Neural networks in civil engineering. II: Systems and application , 1994 .

[27]  TARX models for spikes and antispikes in electricity markets , 2010, 2010 7th International Conference on the European Energy Market.

[28]  U. Rajendra Acharya,et al.  Automated Diagnosis of epilepsy using CWT, HOS and Texture parameters , 2013, Int. J. Neural Syst..

[29]  Mauro Ursino,et al.  A Multi-Layer Neural-Mass Model for Learning Sequences using Theta/Gamma oscillations , 2013, Int. J. Neural Syst..

[30]  Brett A. Story,et al.  A Structural Impairment Detection System Using Competitive Arrays of Artificial Neural Networks , 2014, Comput. Aided Civ. Infrastructure Eng..

[31]  I. Flood,et al.  Neural networks in civil engineering: a review , 2001 .

[32]  Hani Hagras,et al.  Evolving spiking neural network controllers for autonomous robots , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[33]  Peter Tiño,et al.  Minimum Complexity Echo State Network , 2011, IEEE Transactions on Neural Networks.

[34]  Kun-Huang Huarng,et al.  The application of neural networks to forecast fuzzy time series , 2006 .

[35]  Bernard Bobée,et al.  Daily reservoir inflow forecasting using artificial neural networks with stopped training approach , 2000 .

[36]  Milton S. Boyd,et al.  Designing a neural network for forecasting financial and economic time series , 1996, Neurocomputing.

[37]  Stefan Schliebs,et al.  Towards Spatio-Temporal Pattern Recognition Using Evolving Spiking Neural Networks , 2010, ICONIP.

[38]  Bartosz Krawczyk,et al.  Improved Adaptive Splitting and Selection: the Hybrid Training Method of a Classifier Based on a Feature Space Partitioning , 2014, Int. J. Neural Syst..

[39]  Nicolas Brodu.  Quantifying the Effect of Learning on Recurrent Spikin Neurons , 2007, 2007 International Joint Conference on Neural Networks.

[40]  Ran Ginosar,et al.  Adaptive Cardiac Resynchronization Therapy Device Based on Spiking Neurons Architecture and Reinforcement Learning Scheme , 2007, IEEE Transactions on Neural Networks.

[41]  Hongzhe Dai,et al.  An Adaptive Wavelet Frame Neural Network Method for Efficient Reliability Analysis , 2014, Comput. Aided Civ. Infrastructure Eng..

[42]  Nikola K. Kasabov,et al.  Evolving Probabilistic Spiking Neural Networks for Spatio-temporal Pattern Recognition: A Preliminary Study on Moving Object Recognition , 2011, ICONIP.

[43]  Michael Y. Hu,et al.  Neural network forecasting of the British pound/US dol-lar exchange rate , 1998 .

[44]  Ian Flood,et al.  Neural Networks in Civil Engineering. I: Principles and Understanding , 1994 .

[45]  Antoni Morro,et al.  Chaos-Based Mixed Signal Implementation of Spiking Neurons , 2009, Int. J. Neural Syst..

[46]  Hojjat Adeli,et al.  Spiking Neural Networks , 2009, Int. J. Neural Syst..

[47]  Marco Wiering,et al.  Democratic Liquid State Machines for Music Recognition , 2008, Speech, Audio, Image and Biomedical Signal Processing using Neural Networks.

[48]  Yongji Wang,et al.  Modelling of monkey's motor cortical signals by an extended adaptive Liquid State Machine: an integrated procedure from model, identification, experiments, data fitting, to validation , 2011 .

[49]  Dan Ventura,et al.  Spatiotemporal Pattern Recognition via Liquid State Machines , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[50]  Hermann Haken,et al.  Exploring the Brain , 2013 .

[51]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[52]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[53]  Christopher J. Rozell,et al.  Optimal Sparse Approximation with Integrate and Fire Neurons , 2014, Int. J. Neural Syst..

[54]  Silvia Tolu,et al.  Adaptive and Predictive Control of a Simulated Robot arm , 2013, Int. J. Neural Syst..

[55]  Grzegorz M. Wójcik.  Electrical parameters influence on the dynamics of the Hodgkin-Huxley liquid state machine , 2012, Neurocomputing.

[56]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[57]  Farmer,et al.  Predicting chaotic time series. , 1987, Physical review letters.

[58]  Benjamin Schrauwen,et al.  Reservoir-based techniques for speech recognition , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[59]  Henry Markram,et al.  A New Approach towards Vision Suggested by Biologically Realistic Neural Microcircuit Models , 2002, Biologically Motivated Computer Vision.

[60]  Marzuki Khalid,et al.  Nonlinear Identification of a Magneto-Rheological Damper Based on Dynamic Neural Networks , 2014, Comput. Aided Civ. Infrastructure Eng..

[61]  Ilaria Stura,et al.  Double-Layered Models Can Explain Macro and Micro Structure of Human Sleep , 2013, Int. J. Neural Syst..

[62]  Hugo Valadares Siqueira.  Unorganized machines to seasonal streamflow series forecasting , 2013, Int. J. Neural Syst..

[63]  Benjamin Schrauwen,et al.  Defect Detection in Reinforced Concrete Using Random Neural Architectures , 2014, Comput. Aided Civ. Infrastructure Eng..

[64]  Mark Kröll,et al.  Movement Prediction from Real-World Images Using a Liquid State Machine , 2005, IEA/AIE.

[65]  Guido Bugmann,et al.  NEURAL NETWORK DESIGN FOR ENGINEERING APPLICATIONS , 2001 .

[66]  W. Maass,et al.  State-dependent computations: spatiotemporal processing in cortical networks , 2009, Nature Reviews Neuroscience.

[67]  Tatiana O. Blinova,et al.  Analysis of possibility of using neural network to forecast passenger traffic flows in Russia , 2007 .

[68]  Hojjat Adeli,et al.  Improved spiking neural networks for EEG classification and epilepsy and seizure detection , 2007, Integr. Comput. Aided Eng..

[69]  Grzegorz M. Wojcik,et al.  Liquid State Machine Built of Hodgkin-Huxley Neurons , 2004 .

[70]  L. Appeltant,et al.  Information processing using a single dynamical node as complex system , 2011, Nature communications.

[71]  Hojjat Adeli,et al.  A probabilistic neural network for earthquake magnitude prediction , 2009, Neural Networks.

[72]  L Pesquera,et al.  Photonic information processing beyond Turing: an optoelectronic implementation of reservoir computing. , 2012, Optics express.

[73]  Ashwani Kumar,et al.  Electricity price forecasting in deregulated markets: A review and evaluation , 2009 .

[74]  VASSILIS S. KODOGIANNIS,et al.  A Clustering-Based Fuzzy Wavelet Neural Network Model for Short-Term Load Forecasting , 2013, Int. J. Neural Syst..

[75]  Jaakko Hollmén,et al.  Sequential input selection algorithm for long-term prediction of time series , 2008, Neurocomputing.

[76]  Stefan Schliebs,et al.  Spiking Neural Network for On-line Cognitive Activity Classification Based on EEG Data , 2013, ICONIP.

[77]  Benjamin Schrauwen,et al.  Optoelectronic Reservoir Computing , 2011, Scientific Reports.

[78]  Antoni Morro,et al.  Hardware Implementation of Stochastic Spiking Neural Networks , 2012, Int. J. Neural Syst..

[79]  Hualou Liang,et al.  Perceptual suppression revealed by Adaptive Multi-Scale Entropy Analysis of Local Field potential in monkey Visual cortex , 2013, Int. J. Neural Syst..

[80]  Yang Gao,et al.  Structurally Enhanced Incremental Neural Learning for Image Classification with Subgraph Extraction , 2014, Int. J. Neural Syst..

[81]  Kun Wang,et al.  The Application of Liquid State Machines in Robot Path Planning , 2009, J. Comput..

[82]  N. Amjady.  Day-ahead price forecasting of electricity markets by a new fuzzy neural network , 2006, IEEE Transactions on Power Systems.

[83]  Benjamin Schrauwen,et al.  An experimental unification of reservoir computing methods , 2007, Neural Networks.

[84]  J. J. Hopfield,et al.  Pattern recognition computation using action potential timing for stimulus representation , 1995, Nature.

[85]  Sander M. Bohte,et al.  Unsupervised clustering with spiking neurons by sparse temporal coding and multilayer RBF networks , 2002, IEEE Trans. Neural Networks.