A new classification approach for neural networks hardware: from standards chips to embedded systems on chip

The aim of this paper is to propose a new classification approach of artificial neural networks hardware. Our motivation behind this work is justified by the following two arguments: first, during the last two decades a lot of approaches have been proposed for classification of neural networks hardware. However, at present there is not a clear consensus on classification criteria and performances. Second, with the evolution of the microelectronic technology and the design tools and techniques, new artificial neural networks (ANNs) implementations have been proposed, but they are not taken into consideration in the existing classification approaches of ANN hardware. In this paper, we propose a new approach for classification of neural networks hardware. The paper is organized in three parts: in the first part we review most of existing approaches proposed in the literature during the period 1990–2010 and show the advantages and disadvantages of each one. In the second part, we propose a new classification approach that takes into account most of consensual elements in one hand and in the other hand it takes into consideration the evolution of the design technology of integrated circuits and the design techniques. In the third part, we review examples of neural hardware achievements from industrial, academic and research institutions. According to our classification approach, these achievements range from standard chips to VLSI ASICs, FPGA and embedded systems on chip. Finally, we enumerate design issues that are still posed. This could help to give new directions for future research work.

[1]  Massimo A. Sivilotti,et al.  Real-time visual computations using analog CMOS processing arrays , 1987 .

[2]  Andrzej Cichocki,et al.  Neural networks for optimization and signal processing , 1993 .

[3]  Karl Goser,et al.  Systolic Synthesis of Neural Networks , 1990 .

[4]  I. Aleksander,et al.  WISARD·a radical step forward in image recognition , 1984 .

[5]  Fearghal Morgan,et al.  Investigating the Suitability of FPAAs for Evolved Hardware Spiking Neural Networks , 2008, ICES.

[6]  H. C. Card,et al.  CMOS mean field learning , 1991 .

[7]  Andrés Pérez Uribe,et al.  Structure-Adaptable Digital Neural Networks , 1999 .

[8]  Shawki Areibi,et al.  ON THE ARITHMETIC PRECISION FOR IMPLEMENTING BACK-PROPAGATION NETWORKS ON FPGA: A CASE STUDY , 2006 .

[9]  D. Hammerstrom,et al.  A VLSI architecture for high-performance, low-cost, on-chip learning , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  Bertil Svensson,et al.  Using and Designing Massively Parallel Computers for Artificial Neural Neural Networks , 1992, J. Parallel Distributed Comput..

[11]  Tetsuya Yagi,et al.  A Robot Vision System Using a Silicon Retina , 2010, Brain-Inspired Information Technology.

[12]  Reinhard Männer,et al.  Multiprocessor And Memory Architecture Of The Neurocomputer Synapse-1 , 1993, Int. J. Neural Syst..

[13]  S. Tam,et al.  An electrically trainable artificial neural network (ETANN) with 10240 'floating gate' synapses , 1990, International 1989 Joint Conference on Neural Networks.

[14]  S. J. Prange,et al.  Architectures for a Biology-Oriented Neuroemulator , 1991 .

[15]  Simon M. Tam,et al.  Implementation and performance of an analog nonvolatile neural network , 1993 .

[16]  Yuji Sato,et al.  Development of a high-performance general purpose neuro-computer composed of 512 digital neurons , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[17]  Heinrich Klar,et al.  A SIMD/dataflow architecture for a neurocomputer for spike-processing neural networks (NESPINN) , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[18]  A. J. De Groot,et al.  Systolic Implementation Of Neural Network , 1989, Photonics West - Lasers and Applications in Science and Engineering.

[19]  Anne Menendez,et al.  Fish Inspection System Using a Parallel Neural Network Chip and the Image Knowledge Builder Application , 2007, AI Mag..

[20]  M. Weinfeld,et al.  A digital CMOS fully connected neural network with in-circuit learning capability and automatic identification of spurious attractors , 1991 .

[21]  Luis Gómez-Chova,et al.  An IP Core and GUI for Implementing Multilayer Perceptron with a Fuzzy Activation Function on Configurable Logic Devices , 2008, J. Univers. Comput. Sci..

[22]  J.E. Franca,et al.  A mixed-mode architecture for implementation of analog neural networks with digital programmability , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[23]  Joan Cabestany,et al.  The Dynamic Ring Architecture , 1992 .

[24]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[25]  Tomas Nordström,et al.  On-line localized learning systems : Part II - Parallel computer implementation , 1995 .

[26]  Anthony G. Pipe,et al.  Design and FPGA implementation of an embedded real-time biologically plausible spiking neural network processor , 2005, International Conference on Field Programmable Logic and Applications, 2005..

[27]  M. Vellasco,et al.  VLSI architectures for neural networks , 1989, IEEE Micro.

[28]  Michael J. Flynn,et al.  Some Computer Organizations and Their Effectiveness , 1972, IEEE Transactions on Computers.

[29]  Robert David,et al.  Description and practical uses of IBM ZISC036 , 1999, Other Conferences.

[30]  T. Watanabe,et al.  Neural network simulation on a massively parallel cellular array processor: AAP-2 , 1989, International 1989 Joint Conference on Neural Networks.

[31]  José Luis Huertas,et al.  A CMOS analog adaptive BAM with on-chip learning and weight refreshing , 1993, IEEE Trans. Neural Networks.

[32]  Misha Mahowald,et al.  Analog VLSI chip for stereocorrespondence , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[33]  E. Culurciello,et al.  A biomorphic digital image sensor , 2003, IEEE J. Solid State Circuits.

[34]  J. Beichter,et al.  Design of a 1st Generation Neurocomputer , 1991 .

[35]  Edgar Sanchez-Sinencio,et al.  Nonlinear switched capacitor 'neural' networks for optimization problems , 1990 .

[36]  John Wawrzynek,et al.  The design of a neuro-microprocessor , 1993, IEEE Trans. Neural Networks.

[37]  F. Faggin,et al.  Neural network hardware , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[38]  Manfred Glesner,et al.  Neurocomputers: an overview of neural networks in VLSI , 1994 .

[39]  Mark R. DeYong,et al.  The design, fabrication, and test of a new VLSI hybrid analog-digital neural processing element , 1992, IEEE Trans. Neural Networks.

[40]  Sorin Draghici,et al.  Neural Networks in Analog Hardware - Design and Implementation Issues , 2000, Int. J. Neural Syst..

[41]  Philip James-Roxby,et al.  Adapting constant multipliers in a neural network implementation , 2000, Proceedings 2000 IEEE Symposium on Field-Programmable Custom Computing Machines (Cat. No.PR00871).

[42]  Ben J Hicks,et al.  SPIE - The International Society for Optical Engineering , 2001 .

[43]  Hugo de Garis,et al.  The CAM-Brain Machine (CBM): an FPGA-based hardware tool that evolves a 1000 neuron-net circuit module in seconds and updates a 75 million neuron artificial brain for real-time robot control , 2002, Neurocomputing.

[44]  W. Daniel Hillis,et al.  Data parallel algorithms , 1986, CACM.

[45]  S. Bavan,et al.  A simple VLSI architecture for neurocomputing , 1988 .

[46]  V. Peiris,et al.  Implementation of a fully parallel Kohonen map: a mixed analog digital approach , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[47]  Marwan A. Jabri,et al.  Weight perturbation: an optimal architecture and learning technique for analog VLSI feedforward and recurrent multilayer networks , 1992, IEEE Trans. Neural Networks.

[48]  Heinrich Klar,et al.  Digital Neurohardware: Principles and Perspectives , 1998 .

[49]  Alan F. Murray,et al.  Pulse-stream VLSI neural networks mixing analog and digital techniques , 1991, IEEE Trans. Neural Networks.

[50]  Paolo Ienne,et al.  Digital systems for neural networks , 1995, Defense + Commercial Sensing.

[51]  Klaus Schumacher,et al.  VLSI technologies for artificial neural networks , 1989, IEEE Micro.

[52]  Alvise Sartori,et al.  Advances in the design of the TOTEM neurochip , 1997 .

[53]  Miroslav Skrbek,et al.  Fast Neural Network Implementation , 2006 .

[54]  Alan F. Murray,et al.  Asynchronous VLSI neural networks using pulse-stream arithmetic , 1988 .

[55]  G. M. Blair,et al.  PLA design for single-clock CMOS , 1992 .

[56]  Hiroki Matsumoto,et al.  Backpropagation learning in analog T-Model neural network hardware , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[57]  A. Fernandez,et al.  Design of a pipelined hardware architecture for real-time neural network computations , 2002, The 2002 45th Midwest Symposium on Circuits and Systems, 2002. MWSCAS-2002..

[58]  Anton Gunzinger,et al.  Fast neural net simulation with a DSP processor array , 1995, IEEE Trans. Neural Networks.

[59]  Eduardo Sanchez,et al.  Hardware Reconfigurable Neural Networks , 1998, IPPS/SPDP Workshops.

[60]  Benjamin Schrauwen,et al.  Parallel hardware implementation of a broad class of spiking neurons using serial arithmetic , 2006, ESANN.

[61]  Paul E. Hasler,et al.  Single Transistor Learning Synapses , 1994, NIPS.

[62]  D. Hammerstrom,et al.  Distributing back propagation networks over the Intel iPSC/860 hypercube , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[63]  Jagath C. Rajapakse,et al.  FPGA Implementations of Neural Networks , 2006 .

[64]  Niels Mache,et al.  Recent developments of the SNNS neural network simulator , 1991 .

[65]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[66]  Paolo Ienne,et al.  GENES IV: A bit-serial processing element for a multi-model neural-network accelerator , 1993, J. VLSI Signal Process..

[67]  Douglas S. Reeves,et al.  The TInMANN VLSI chip , 1992, IEEE Trans. Neural Networks.

[68]  M. Duranton,et al.  Learning on VLSI: a general purpose digital neurochip , 1989, International 1989 Joint Conference on Neural Networks.

[69]  Mohammed Ismail,et al.  Analog VLSI Implementation of Neural Systems , 2011, The Kluwer International Series in Engineering and Computer Science.

[70]  André DeHon,et al.  The Density Advantage of Configurable Computing , 2000, Computer.

[71]  W. E. Blanz,et al.  GANGLION-a fast field-programmable gate array implementation of a connectionist classifier , 1992 .

[72]  Karl Mathia,et al.  Learning flight control and LoFLYTE , 1995, Proceedings of WESCON'95.

[73]  Hal McCartor,et al.  Back Propagation Implementation on the Adaptive Solutions CNAPS Neurocomputer Chip , 1990, NIPS 1990.

[74]  David S. Touretzky,et al.  Advances in neural information processing systems 2 , 1989 .

[75]  Mikael Taveniku,et al.  A reconfigurable SIMD computer for artificial neural networks , 1995 .

[76]  Nouma Izeboudjen,et al.  A reuse oriented design methodology for artificial neural networks implementation , 1999, Twelfth Annual IEEE International ASIC/SOC Conference (Cat. No.99TH8454).

[77]  Johannes Schemmel,et al.  Speeding up Hardware Evolution: A Coprocessor for Evolutionary Algorithms , 2003, ICES.

[78]  A. Masaki,et al.  A wafer scale integration neural network utilizing completely digital circuits , 1989, International 1989 Joint Conference on Neural Networks.

[79]  Ulrich Rückert,et al.  Implementation of artificial neural networks on a reconfigurable hardware accelerator , 2002, Proceedings 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing.

[80]  Dimitris Anastassiou,et al.  Switched-capacitor neural networks , 1987 .

[81]  Chung-Yu Wu,et al.  Analog electronic cochlea design using multiplexing switched-capacitor circuits , 1996, IEEE Trans. Neural Networks.

[82]  H. T. Kung,et al.  The Warp Computer: Architecture, Implementation, and Performance , 1987, IEEE Transactions on Computers.

[83]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[84]  Jonathan S. Kane,et al.  POPART: partial optical implementation of adaptive resonance theory 2 , 1993, IEEE Trans. Neural Networks.

[85]  Tetsuya Yagi,et al.  An analog VLSI chip emulating sustained and transient response channels of the vertebrate retina , 2003, IEEE Trans. Neural Networks.

[86]  Denis F. Wolf,et al.  USING EMBEDDED PROCESSORS IN HARDWARE MODELS OF ARTIFICIAL NEURAL NETWORKS , 2001 .

[87]  Howard B. Demuth,et al.  Neutral network toolbox for use with Matlab , 1995 .

[88]  Tetsuro Itakura,et al.  Neuro chips with on-chip back-propagation and/or Hebbian learning , 1992 .

[89]  James M. Bower,et al.  The Book of GENESIS , 1994, Springer New York.

[90]  Gregory D. Peterson,et al.  Evolvable Block-Based Neural Network Design for Applications in Dynamic Environments , 2010, VLSI Design.

[91]  Valeriu Beiu,et al.  Digital integrated circuit implementations , 1997 .

[92]  William A. Fisher,et al.  A programmable analog neural network processor , 1991, IEEE Trans. Neural Networks.

[93]  P. Ienne,et al.  MANTRA I: a systolic neuro-computer , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[94]  Andres Upegui,et al.  An FPGA platform for on-line topology exploration of spiking neural networks , 2005, Microprocess. Microsystems.

[95]  Alan F. Murray,et al.  A Novel Computational and Signalling Method for VLSI Neural Networks , 1987, ESSCIRC '87: 13th European Solid-State Circuits Conference.

[96]  Veljko Milutinovic,et al.  Neural Networks: Concepts, Applications, and Implementations , 1991 .

[97]  Jeff A. Bilmes,et al.  The RAP: a ring array processor for layered network calculations , 1990, [1990] Proceedings of the International Conference on Application Specific Array Processors.

[98]  J. L. Holt,et al.  Back propagation simulations using limited precision calculations , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[99]  Bruce Denby,et al.  The Use of Neural Networks in High-Energy Physics , 1993, Neural Computation.

[100]  George J. Milne,et al.  Towards an FPGA based reconfigurable computing environment for neural network implementations , 1999 .

[101]  Hideo Mitsui,et al.  ANN accelerator by parallel processor based on DSP , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[102]  M. Weinfeld,et al.  A digital CMOS fully connected neural network with in-circuit learning capability and automatic identification of spurious attractors , 1991, Euro ASIC '91.

[103]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[104]  Jos Nijhuis,et al.  Automatic generation of VHDL code for neural applications , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[105]  Ulrich Rückert,et al.  VLSI Design of Neural Networks , 1990 .

[106]  Mohamed I. Elmasry,et al.  VLSI Artificial Neural Networks Engineering , 1994 .

[107]  Peter J. Bentley,et al.  An FPGA-based model suitable for evolution and development of spiking neural networks , 2008, ESANN.

[108]  Peter Christy,et al.  Software to support massively parallel computing on the MasPar MP-1 , 1990, Digest of Papers Compcon Spring '90. Thirty-Fifth IEEE Computer Society International Conference on Intellectual Leverage.

[109]  Vipan Kakkar,et al.  Comparative Study on Analog and Digital Neural Networks , 2009 .

[110]  Brad Hutchings,et al.  RRANN: the run-time reconfiguration artificial neural network , 1994, Proceedings of IEEE Custom Integrated Circuits Conference - CICC '94.

[111]  D. J. Hunt.  AMT DAP—a processor array in a workstation environment , 1989 .

[112]  Toshiyuki Furuta,et al.  Neural network LSI chip with on-chip learning , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.