An Overview of Neuromorphic Computing for Artificial Intelligence Enabled Hardware-Based Hopfield Neural Network

Compared with von Neumann’s computer architecture, neuromorphic systems offer more unique and novel solutions to the artificial intelligence discipline. Inspired by biology, this novel system has implemented the theory of human brain modeling by connecting feigned neurons and synapses to reveal the new neuroscience concepts. Many researchers have vastly invested in neuro-inspired models, algorithms, learning approaches, operation systems for the exploration of the neuromorphic system and have implemented many corresponding applications. Recently, some researchers have demonstrated the capabilities of Hopfield algorithms in some large-scale notable hardware projects and seen significant progression. This paper presents a comprehensive review and focuses extensively on the Hopfield algorithm’s model and its potential advancement in new research applications. Towards the end, we conclude with a broad discussion and a viable plan for the latest application prospects to facilitate developers with a better understanding of the aforementioned model in accordance to build their own artificial intelligence projects.

[1]  Le Song,et al.  Joint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural Networks , 2017, ArXiv.

[2]  Xiaolin Hu,et al.  Training the Hopfield Neural Network for Classification Using a STDP-Like Rule , 2017, ICONIP.

[3]  Yu Chen,et al.  Polymer memristor for information storage and neuromorphic applications , 2014 .

[4]  J. Keeler.  Comparison Between Kanerva's SDM and Hopfield-Type Neural Networks , 1988, Cogn. Sci..

[5]  Simona Cocco,et al.  Statistical physics and representations in real and artificial neural networks , 2017, Physica A: Statistical Mechanics and its Applications.

[6]  Volker Schmid,et al.  Pattern Recognition and Signal Analysis in Medical Imaging , 2003 .

[7]  Giacomo Indiveri,et al.  A spiking implementation of the lamprey's Central Pattern Generator in neuromorphic VLSI , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[8]  Don Monroe,et al.  Neuromorphic computing gets ready for the (really) big time , 2014, CACM.

[9]  Leon O. Chua,et al.  Cellular neural networks: applications , 1988 .

[10]  X. Miao,et al.  Activity-Dependent Synaptic Plasticity of a Chalcogenide Electronic Synapse for Neuromorphic Systems , 2014, Scientific Reports.

[11]  Satoshi Matsuda,et al.  "Optimal" Hopfield network for combinatorial optimization with linear cost function , 1998, IEEE Trans. Neural Networks.

[12]  J. Kotaleski,et al.  Modelling the molecular mechanisms of synaptic plasticity using systems biology approaches , 2010, Nature Reviews Neuroscience.

[13]  Berndt Müller,et al.  Neural networks: an introduction , 1990 .

[14]  Fernando Niño,et al.  Classification of Natural Language Sentences using Neural Networks , 2003, FLAIRS.

[15]  Patrick van der Smagt,et al.  Introduction to neural networks , 1995, The Lancet.

[16]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[17]  Mykola Pechenizkiy,et al.  Evolving Plasticity for Autonomous Learning under Changing Environmental Conditions , 2019, Evolutionary Computation.

[18]  Ye Zhang,et al.  Study on the Capacity of Hopfield Neural Networks , 2008 .

[19]  Santosh S. Venkatesh,et al.  The capacity of the Hopfield associative memory , 1987, IEEE Trans. Inf. Theory.

[20]  Toshiyuki Yamane,et al.  Recent Advances in Physical Reservoir Computing: A Review , 2018, Neural Networks.

[21]  Yau-Hwang Kuo,et al.  A fuzzy neural network model and its hardware implementation , 1993, IEEE Trans. Fuzzy Syst..

[22]  Peter Wittek,et al.  Quantum Machine Learning: What Quantum Computing Means to Data Mining , 2014 .

[23]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[24]  Zekeriya Uykan,et al.  Continuous-time Hopfield neural network-based optimized solution to 2-channel allocation problem , 2015 .

[25]  A. Thomas,et al.  The Memristive Magnetic Tunnel Junction as a Nanoscopic Synapse‐Neuron System , 2012, Advanced materials.

[26]  Tomasz Szandala.  Comparison of Different Learning Algorithms for Pattern Recognition with Hopfield's Neural Network , 2015, BICA.

[27]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[28]  Okan K. Ersoy,et al.  A MULTISTAGE APPROACH TO THE HOPFIELD MODEL FOR BI-LEVEL IMAGE RESTORATION , 1995 .

[29]  Sorin Draghici,et al.  Neural Networks in Analog Hardware - Design and Implementation Issues , 2000, Int. J. Neural Syst..

[30]  Mohamad A. Akra.  On the Analysis of The Hopfield Network: A Geometric Approach. , 1988 .

[31]  Pengfei Shi,et al.  An Algorithm for License Plate Recognition Applied to Intelligent Transportation System , 2011, IEEE Transactions on Intelligent Transportation Systems.

[32]  Timothée Masquelier,et al.  Deep Learning in Spiking Neural Networks , 2018, Neural Networks.

[33]  Yoshiaki Watanabe,et al.  Solving optimization problems by using a Hopfield neural network and genetic algorithm combination , 1998, Systems and Computers in Japan.

[34]  Yash Pal Singh,et al.  Analysis of Hopfield Autoassociative Memory in the Character Recognition , 2010 .

[35]  B. Bavarian,et al.  Introduction to neural networks for intelligent control , 1988, IEEE Control Systems Magazine.

[36]  Giacomo Indiveri,et al.  A Scalable Multicore Architecture With Heterogeneous Memory Structures for Dynamic Neuromorphic Asynchronous Processors (DYNAPs) , 2017, IEEE Transactions on Biomedical Circuits and Systems.

[37]  Yan Lou,et al.  Using Auto-Associative Neural Networks for Signal Recognition Technology on Sky Screen , 2014 .

[38]  Zili Liu,et al.  Limited Top-Down Influence from Recognition to Same-Different Matching of Chinese Characters , 2016, PloS one.

[39]  Hong Wang,et al.  Loihi: A Neuromorphic Manycore Processor with On-Chip Learning , 2018, IEEE Micro.

[40]  E.D. Di Claudio,et al.  Car plate recognition by neural networks and image processing , 1998, ISCAS '98. Proceedings of the 1998 IEEE International Symposium on Circuits and Systems (Cat. No.98CH36187).

[41]  N. Brunel,et al.  Calcium-based plasticity model explains sensitivity of synaptic changes to spike pattern, rate, and dendritic location , 2012, Proceedings of the National Academy of Sciences.

[42]  Jianhui Zhao,et al.  Vacancy-Induced Synaptic Behavior in 2D WS2 Nanosheet-Based Memristor for Low-Power Neuromorphic Computing. , 2019, Small.

[43]  James L. McClelland.  Running head : HEBBIAN LEARNING How Far Can You Go with Hebbian Learning , and When Does it Lead you Astray ? , 2005 .

[44]  Giancarlo Ruocco,et al.  On the Maximum Storage Capacity of the Hopfield Model , 2017, Frontiers Comput. Neurosci..

[45]  Kaushik Roy,et al.  Towards spike-based machine intelligence with neuromorphic computing , 2019, Nature.

[46]  Sargur N. Srihari,et al.  Recognition of handwritten and machine-printed text for postal address interpretation , 1993, Pattern Recognit. Lett..

[47]  Laura Cantini,et al.  Hope4Genes: a Hopfield-like class prediction algorithm for transcriptomic data , 2019, Scientific Reports.

[48]  Wulfram Gerstner,et al.  SPIKING NEURON MODELS Single Neurons , Populations , Plasticity , 2002 .

[49]  Kwabena Boahen,et al.  Silicon neurons that inhibit to synchronize , 2006, 2006 IEEE International Symposium on Circuits and Systems.

[50]  Ivan K. Schuller,et al.  Neuromorphic Computing – From Materials Research to Systems Architecture Roundtable , 2015 .

[51]  Qingyun Ma,et al.  Bursting Hodgkin–Huxley model-based ultra-low-power neuromimetic silicon neuron , 2012 .

[52]  Xinhua Zhuang,et al.  Better learning for bidirectional associative memory , 1993, Neural Networks.

[53]  Narotam Singh,et al.  Low-Resolution Image Recognition Using Cloud Hopfield Neural Network , 2018 .

[54]  James A. Hendler,et al.  Efficient Classification of Supercomputer Failures Using Neuromorphic Computing , 2018, 2018 IEEE Symposium Series on Computational Intelligence (SSCI).

[55]  Ursula Challita,et al.  Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial , 2017, IEEE Communications Surveys & Tutorials.

[56]  Sina Balkir,et al.  ANNSyS: an Analog Neural Network Synthesis System , 1999, Neural Networks.

[57]  Benoit Corraze,et al.  Control of resistive switching in AM4Q8 narrow gap Mott insulators: A first step towards neuromorphic applications , 2015 .

[58]  Madan M. Gupta,et al.  Static and Dynamic Neural Networks: From Fundamentals to Advanced Theory , 2003 .

[59]  D. Dutta Majumder,et al.  Application of Hopfield neural networks and canonical perspectives to recognize and locate partially occluded 3-D objects , 1994, Pattern Recognit. Lett..

[60]  John J. Hopfield,et al.  Simple 'neural' optimization networks: An A/D converter, signal decision circuit, and a linear programming circuit , 1986 .

[61]  Terence D. Sanger,et al.  Optimal unsupervised learning in a single-layer linear feedforward neural network , 1989, Neural Networks.

[62]  James H. Garrett,et al.  Use of neural networks in detection of structural damage , 1992 .

[63]  Philippe Hurat,et al.  A VLSI Systolic Array Dedicated to Hopfield Neural Network , 1989 .

[64]  Farinaz Koushanfar,et al.  CAMsure: Secure Content-Addressable Memory for Approximate Search , 2017, ACM Trans. Embed. Comput. Syst..

[65]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[66]  Karthikeyan Sankaralingam,et al.  Power struggles: Revisiting the RISC vs. CISC debate on contemporary ARM and x86 architectures , 2013, 2013 IEEE 19th International Symposium on High Performance Computer Architecture (HPCA).

[67]  Fernando Corinto,et al.  Memristor cellular automata through belief propagation inspired algorithm , 2015, 2015 International SoC Design Conference (ISOCC).

[68]  Constantin Virgil Negoita.  Cybernetics and Applied Systems , 1992 .

[69]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[70]  Lei Wang,et al.  Recent Advances on Neuromorphic Systems Using Phase-Change Materials , 2017, Nanoscale Research Letters.

[71]  J. Knott.  The organization of behavior: A neuropsychological theory , 1951 .

[72]  Narayan Srinivasa,et al.  Low-Power Neuromorphic Hardware for Signal Processing Applications: A review of architectural and system-level design approaches , 2019, IEEE Signal Processing Magazine.

[73]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[74]  Hua Yang,et al.  An optimization routing protocol for FANETs , 2019, EURASIP Journal on Wireless Communications and Networking.

[75]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[76]  Alan F. Murray,et al.  Real-Time Autonomous Robot Navigation Using VLSI Neural Networks , 1990, NIPS.

[77]  Kwabena Boahen,et al.  Learning in Silicon: Timing is Everything , 2005, NIPS.

[78]  Emre Neftci,et al.  Stochastic neuromorphic learning machines for weakly labeled data , 2016, 2016 IEEE 34th International Conference on Computer Design (ICCD).

[79]  M. PADMANABAN,et al.  Handwritten Character Recognition using Conditional Probabilities , 2006 .

[80]  Martin A. Riedmiller,et al.  Advanced supervised learning in multi-layer perceptrons — From backpropagation to adaptive learning algorithms , 1994 .

[81]  Lambert Spaanenburg,et al.  Car license plate recognition with neural networks and fuzzy logic , 1995, Proceedings of ICNN'95 - International Conference on Neural Networks.

[82]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[83]  Robert H. Riffenburgh,et al.  Linear Discriminant Analysis , 1960 .

[84]  E. Capaldi,et al.  The organization of behavior. , 1992, Journal of applied behavior analysis.

[85]  Amr A. Adly,et al.  Efficient vector hysteresis modeling using rotationally coupled step functions , 2012 .

[86]  Alex Pappachen James,et al.  Level-shifted neural encoded analog-to-digital converter , 2017, 2017 24th IEEE International Conference on Electronics, Circuits and Systems (ICECS).

[87]  Ue-Pyng Wen,et al.  A review of Hopfield neural networks for solving mathematical programming problems , 2009, Eur. J. Oper. Res..

[88]  Li Zhi-jun.  Pattern Recogition Based on Hopfield Neural Network , 2005 .

[89]  Marcos Aurélio Batista,et al.  Images segmentation using a modified Hopfield artificial neural network , 2018 .

[90]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[91]  Somesh Kumar,et al.  Implementation of Hopfield Neural Network for its Capacity with Finger Print Images , 2016 .

[92]  Derek Abbott,et al.  Digital Multiplierless Realization of Two-Coupled Biological Hindmarsh–Rose Neuron Model , 2016, IEEE Transactions on Circuits and Systems II: Express Briefs.

[93]  Vishal Saxena,et al.  Towards Neuromorphic Learning Machines Using Emerging Memory Devices with Brain-Like Energy Efficiency , 2018, Journal of Low Power Electronics and Applications.

[94]  Giacomo Indiveri,et al.  Memory and Information Processing in Neuromorphic Systems , 2015, Proceedings of the IEEE.

[95]  Simon Osindero,et al.  Meta-Learning Deep Energy-Based Memory Models , 2020, ICLR.

[96]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.

[97]  Carver A. Mead,et al.  Neuromorphic electronic systems , 1990, Proc. IEEE.

[98]  Manish Kumar.  Large-scale neuromorphic computing systems , 2016 .

[99]  Robert J. Marks,et al.  An adaptively trained neural network , 1991, IEEE Trans. Neural Networks.

[100]  Yide Ma,et al.  Review of pulse-coupled neural networks , 2010, Image Vis. Comput..

[101]  Valery Moreno Vega,et al.  Fault Diagnosis with Missing Data Based on Hopfield Neural Networks , 2016 .

[102]  Giuseppe Di Modica,et al.  IoT fault management in cloud/fog environments , 2019, IOT.

[103]  F. Grassia,et al.  Silicon neuron: digital hardware implementation of the quartic model , 2014, Artificial Life and Robotics.

[104]  Ioannis Anagnostopoulos,et al.  A License Plate-Recognition Algorithm for Intelligent Transportation System Applications , 2006, IEEE Transactions on Intelligent Transportation Systems.

[105]  Gürsel Serpen,et al.  Hopfield Network as Static Optimizer: Learning the Weights and Eliminating the Guesswork , 2008, Neural Processing Letters.

[106]  S Y Lee,et al.  Optical implementation of the Hopfield model for two-dimensional associative memory. , 1988, Optics letters.

[107]  Aggelos K. Katsaggelos,et al.  Image restoration using a modified Hopfield network , 1992, IEEE Trans. Image Process..

[108]  Multiplying two numbers together in your head is a difficult task if you did not learn multiplication tables as a child. On the face of it, this is somewhat surprising given the remarkable power of the brain to perform , 2010 .

[109]  Giacomo Indiveri,et al.  Computation in neuromorphic analog VLSI systems: Lecture WS 2001/2002 , 2002 .

[110]  Li Rong,et al.  A new water quality evaluation model based on simplified Hopfield neural network , 2015, 2015 34th Chinese Control Conference (CCC).

[111]  Larry D. Pyeatt.  Modern Assembly Language Programming with the ARM Processor , 2016 .

[112]  Gamal A. Elnashar,et al.  Dynamical Nonlinear Neural Networks with Perturbations Modeling and Global Robust Stability Analysis , 2014 .

[113]  Mohd. Samar Ansari,et al.  Voltage-Mode Neural Network for the Solution of Linear Equations , 2014 .

[114]  Wenxi Lu,et al.  Application of Artificial Neural Network in Environmental Water Quality Assessment , 2013 .

[115]  Hui Yang,et al.  Efficient Hybrid Multi-Faults Location Based on Hopfield Neural Network in 5G Coexisting Radio and Optical Wireless Networks , 2019, IEEE Transactions on Cognitive Communications and Networking.

[116]  Bing J. Sheu,et al.  Parallel digital image restoration using adaptive VLSI neural chips , 1990, Proceedings., 1990 IEEE International Conference on Computer Design: VLSI in Computers and Processors.

[117]  G. Kavitha,et al.  Recalling of Images using Hopfield Neural Network Model , 2011, ArXiv.

[118]  Francesca Mastrogiuseppe,et al.  A Geometrical Analysis of Global Stability in Trained Feedback Networks , 2019, Neural Computation.

[119]  Neha Sahu,et al.  NEURAL NETWORK BASED APPROACH FOR RECOGNITION FOR DEVANAGIRI CHARACTERS , 2014 .

[120]  K. Jeffery,et al.  Modifiable neuronal connections: an overview for psychiatrists. , 1997, The American journal of psychiatry.

[121]  Somesh Kumar,et al.  Performance evaluation of Hopfield neural networks for overlapped English characters by using genetic algorithms , 2011, Int. J. Hybrid Intell. Syst..

[122]  Sompolinsky,et al.  Storing infinite numbers of patterns in a spin-glass model of neural networks. , 1985, Physical review letters.

[123]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[124]  M. Omair Ahmad,et al.  Hopfield network-based image retrieval using re-ranking and voting , 2017, 2017 IEEE 30th Canadian Conference on Electrical and Computer Engineering (CCECE).

[125]  Pooja Yadav,et al.  Enhancing Performance of Devanagari Script Recognition using Hopfield ANN , 2016 .

[126]  Susmita Mohapatra.  Pattern Recall Analysis of the Hopfield Neural Network with a Genetic Algorithm , 2019 .

[127]  Stephen S. Yau,et al.  Associative Processor Architecture—a Survey , 1977, CSUR.

[128]  Herbert Jaeger,et al.  Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns , 2017, J. Mach. Learn. Res..

[129]  S. Ambrogio,et al.  Emerging neuromorphic devices , 2019, Nanotechnology.

[130]  Shaikh Abdul Hannan,et al.  AN OVERVIEW AND APPLICATIONS OF OPTICAL CHARACTER RECOGNITION , 2014 .

[131]  Paul Hasler,et al.  VLSI neural systems and circuits , 1990, Ninth Annual International Phoenix Conference on Computers and Communications. 1990 Conference Proceedings.

[132]  Teruyoshi Washizawa.  Application of Hopfield network to saccades , 1993, IEEE Trans. Neural Networks.

[133]  X. Miao,et al.  Ultrafast Synaptic Events in a Chalcogenide Memristor , 2013, Scientific Reports.

[134]  A. Galves,et al.  Infinite Systems of Interacting Chains with Memory of Variable Length—A Stochastic Model for Biological Neural Nets , 2012, 1212.5505.

[135]  Günther Palm,et al.  Neural associative memories and sparse coding , 2013, Neural Networks.

[136]  Catherine D. Schuman,et al.  A Survey of Neuromorphic Computing and Neural Networks in Hardware , 2017, ArXiv.

[137]  Wansheng Tang,et al.  Analysis and design of asymmetric Hopfield networks with discrete-time dynamics , 2010, Biological Cybernetics.

[138]  Sumio Hosaka,et al.  Associative memory realized by a reconfigurable memristive Hopfield neural network , 2015, Nature Communications.

[139]  Johannes Schemmel,et al.  A comprehensive workflow for general-purpose neural modeling with highly configurable neuromorphic hardware systems , 2010, Biological Cybernetics.

[140]  Hortensia Mecha,et al.  Hardware implementation of a fault-tolerant Hopfield Neural Network on FPGAs , 2016, Neurocomputing.

[141]  Ethan S. Bromberg-Martin,et al.  Dopamine in Motivational Control: Rewarding, Aversive, and Alerting , 2010, Neuron.

[142]  John H. Holmes,et al.  Knowledge Discovery in Biomedical Data: Theory and Methods , 2014 .

[143]  Munish Kumar,et al.  k-nearest neighbor based offline handwritten Gurmukhi character recognition , 2011, 2011 International Conference on Image Information Processing.

[144]  Yu Xie,et al.  Convergence of discrete delayed Hopfield neural networks , 2009, Comput. Math. Appl..

[145]  Yan Zhu,et al.  Computerized tumor boundary detection using a Hopfield neural network , 1997, IEEE Transactions on Medical Imaging.

[146]  Fatih A. Unal.  Temporal Pattern Matching Using an Artificial Neural Network , 1998 .

[147]  Jie Zhang,et al.  Hopfield Neural Network-based Fault Location in Wireless and Optical Networks for Smart City IoT , 2019, 2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC).

[148]  R. Douglas,et al.  Event-Based Neuromorphic Systems , 2015 .

[149]  Renu Dhir,et al.  Use of Gabor Filters for Recognition of Handwritten Gurmukhi Character , 2012 .

[150]  David J. Evans,et al.  A system-level fault diagnosis algorithm based on preprocessing and parallel Hopfield neural network , 2003 .