Fifty years of Electronic Hardware Implementations of First and Higher Order Neural Networks

This chapter celebrates 50 years of first and higher order neural network (HONN) implementations in terms of the physical layout and structure of electronic hardware, which offers high speed, low latency, compact, low cost, low power, mass produced systems. Low latency is essential for practical applications in real time control for which software implementations running on CPUs are too slow. The literature review chapter traces the chronological development of electronic neural networks (ENN) discussing selected papers in detail from analog electronic hardware, through probabilistic RAM, generalizing RAM, custom silicon Very Large Scale Integrated (VLSI) circuit, Neuromorphic chips, pulse stream interconnected neurons to Application Specific Integrated circuits (ASICs) and Zero Instruction Set Chips (ZISCs). Reconfigurable Field Programmable Gate Arrays (FPGAs) are given particular attention as the most recent generation incorporate Digital Signal Processing (DSP) units to provide full System on Chip (SoC) capability offering the possibility of real-time, on-line and on-chip learning. © 2010, IGI Global.

[1]  Michael Fairhurst,et al.  Pattern learning in humans and electronic learning nets , 1970 .

[2]  John G. Taylor,et al.  Generalization in probabilistic RAM nets , 1993, IEEE Trans. Neural Networks.

[3]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[4]  Colin Giles,et al.  Learning, invariance, and generalization in high-order neural networks. , 1987, Applied optics.

[5]  Michael Fairhurst,et al.  Natural pattern clustering in digital learning nets , 1971 .

[6]  Ibrahim I. Esat,et al.  A curvilinear snake arm robot with gripper-axis fibre-optic image processor feedback , 1983, Robotica.

[7]  C. L. Giles,et al.  Machine learning using higher order correlation networks , 1986 .

[8]  Bernard Widrow,et al.  Sensitivity of feedforward neural networks to weight errors , 1990, IEEE Trans. Neural Networks.

[9]  E. Stamos,et al.  Similarity suppression algorithm for designing pattern discrimination filters , 2002 .

[10]  Alan F. Murray,et al.  Programmable Analog Pulse-Firing Neural Networks , 1988, NIPS.

[11]  Leon O. Chua,et al.  Programmable analogue vlsi cnn chip with local digital logic , 1992, Int. J. Circuit Theory Appl..

[12]  Seul Jung,et al.  Hardware Implementation of a Real-Time Neural Network Controller With a DSP and an FPGA for Nonlinear Systems , 2007, IEEE Transactions on Industrial Electronics.

[13]  I. Aleksander.  Brain cell to microcircuit , 1970 .

[14]  Rafael Gadea Gironés,et al.  FPGA Implementation of a Pipelined On-Line Backpropagation , 2005, J. VLSI Signal Process..

[15]  W. K. Taylor,et al.  Effects of border variations due to spatial quantisation on binary-image template matching , 1982 .

[16]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[17]  Fan Yang,et al.  Implementation of an RBF neural network on embedded systems: real-time face tracking and identity verification , 2003, IEEE Trans. Neural Networks.

[18]  Igor Aleksander,et al.  Digital general neural units with controlled transition probabilities , 1996 .

[19]  W. K. Taylor,et al.  A microprocessor-controlled real-time image processor , 1983 .

[20]  I. Aleksander,et al.  Improving the generalisation of the N-tuple classifier using the effective VC dimension , 1996 .

[21]  Christopher M. Bishop,et al.  Machines that learn , 2005 .

[22]  Geoffrey G. Lockwood,et al.  Predicting the behaviour of G-RAM networks , 2003, Neural Networks.

[23]  W. K. Taylor.  Machine learning and recognition of faces , 1967 .

[24]  M. B. Herscher,et al.  Functional Electronic Model of the Frog Retina , 1963, IEEE Transactions on Military Electronics.

[25]  Simon M. Tam,et al.  Implementation and performance of an analog nonvolatile neural network , 1993 .

[26]  Igor Aleksander,et al.  Pattern-recognition properties of r.a.m./r.o.m. arrays , 1977 .

[27]  M. R. Sweet,et al.  Radiation exposure effects on the performance of an electrically trainable artificial neural network (ETANN) , 1993 .

[28]  Amnon Yariv,et al.  Semiparallel microelectronic implementation of neural network models using CCD technology , 1987 .

[29]  Alan F. Murray,et al.  Asynchronous arithmetic for VLSI neural systems , 1987 .

[30]  Ammar Belatreche,et al.  Challenges for large-scale implementations of spiking neural networks on FPGAs , 2007, Neurocomputing.

[31]  Tyge Greibrokk,et al.  Editorial: Multidimensional confusion? , 2006 .

[32]  Alan F. Murray,et al.  Asynchronous VLSI neural networks using pulse-stream arithmetic , 1988 .

[33]  W. K. Taylor.  Cortico-thalamic organization and memory , 1964, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[34]  W K Taylor,et al.  A comparative study of electronic and neural networks involved in pattern recognition. , 1973, Journal of theoretical biology.

[35]  I. Aleksander,et al.  Microcircuit learning nets: some tests with handwritten numerals , 1968 .

[36]  Igor Aleksander,et al.  General neural unit: retrieval performance , 1991 .

[37]  J. E. Midwinter,et al.  Correlating matched-filter model for analysis and optimisation of neural networks , 1989 .

[38]  E. H. Mamdani,et al.  Microcircuit learning nets: improved recognition by means of pattern feedback , 1968 .

[39]  Richard F. Lyon,et al.  An analog electronic cochlea , 1988, IEEE Trans. Acoust. Speech Signal Process..

[40]  John Taylor,et al.  Design, implementation and evaluation of a high-speed integrated Hamming neural classifier , 1994 .

[41]  W. K. Taylor,et al.  A parallel processing model for the study of possible pattern recognition mechanisms in the brain , 1976, Pattern Recognit..

[42]  I. Aleksander.  Microcircuit learning nets: Hamming-distance behaviour , 1970 .

[43]  L. Yunxi.  Application of decision feedback equalisers based on multilayer neural network structures to compensate , 2007 .

[44]  Dimitris Anastassiou,et al.  Switched-capacitor neural networks , 1987 .

[45]  Leon O. Chua,et al.  The CNN paradigm , 1993 .