FPNA: Concepts and Properties

Neural networks are usually considered as naturally parallel computing models. But the number of operators and the complex connection graph of standard neural models can not be handled by digital hardware devices. Though programmable digital hardware now stand as a real opportunity for flexible hardware implementations of neural networks, many area and topology problems arise when standard neural models are implemented onto programmable circuits such as FPGAs, so that the fast FPGA technology improvements can not be fully exploited. The theoretical and practical framework first introduced in [21] reconciles simple hardware topologies with complex neural architectures, thanks to some configurable hardware principles applied to neural computation: Field Programmable Neural Arrays (FPNA) lead to powerful neural architectures that are easy to map onto FPGAs, by means of a simplified topology and an original data exchange scheme. This two-chapter study gathers the different results that have been published about the FPNA concept, as well as some unpublished ones. This first part focuses on definitions and theoretical aspects. Starting from a general two-level definition of FPNAs, all proposed computation schemes are together described and compared. Their correctness and partial equivalence is justified. The computational power of FPNA-based neural networks is characterized through the concept of underparameterized convolutions.

[1]  Michael Gschwind,et al.  A Fast FPGA Implementation of a General Purpose Neuron , 1994, FPL.

[2]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[3]  Bernard Girau,et al.  MLP computing and learning on FPGA using on-line arithmetic , 1999 .

[4]  Brad Hutchings,et al.  RRANN: a hardware implementation of the backpropagation algorithm using reconfigurable FPGAs , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[5]  Narasimhan Sundararajan,et al.  Parallel implementation of backpropagation neural networks on a heterogeneous array of transputers , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[6]  Luigi Carro,et al.  FPGA architecture comparison for non-conventional signal processing , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[7]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[8]  Hélène Paugam-Moisy.  Optimal Speedup Conditions for a Parallel Back-Propagation Algorithm , 1992, CONPAR.

[9]  G. Horvath,et al.  A full-parallel digital implementation for pre-trained NNs , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[10]  J. Shawe-Taylor,et al.  A stochastic neural architecture that exploits dynamically reconfigurable FPGAs , 1993, [1993] Proceedings IEEE Workshop on FPGAs for Custom Computing Machines.

[11]  Soo-Young Lee,et al.  Effects of multiplier output offsets on on-chip learning for analog neuro-chips , 2004, Neural Processing Letters.

[12]  Ralf Östermark.  A flexible multicomputer algorithm for artificial neural networks , 1996, Neural Networks.

[13]  Yutaka Maeda,et al.  FPGA implementation of a pulse density neural network using simultaneous perturbation , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[14]  Bernard Girau.  Conciliating Connectionism and Parallel Digital Hardware , 2000, Scalable Comput. Pract. Exp..

[15]  Bernard Girau,et al.  Generic Back-Propagation in Arbitrary FeedForward Neural Networks , 1995, ICANNGA.

[16]  Karl Goser,et al.  Exponential Hebbian On-Line Learning Implemented in FPGAs , 1996, ICANN.

[17]  John Wawrzynek,et al.  The design of a neuro-microprocessor , 1993, IEEE Trans. Neural Networks.

[18]  Alfred Strey,et al.  Design of a low-cost and high-speed neurocomputer system , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[19]  Jean-Luc Gaudiot,et al.  Parallel Implementations of Neural Networks , 1993, Int. J. Artif. Intell. Tools.

[20]  Bernard Girau.  Building a 2D-compatible multilayer neural network , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[21]  Karl Goser,et al.  Short- and Long-Term Dynamics in a Stochastic Pulse Stream Neuron Implemented in FPGA , 1997, ICANN.

[22]  Ronald S. Gyurcsik,et al.  Toward a general-purpose analog VLSI neural network with on-chip learning , 1997, IEEE Trans. Neural Networks.

[23]  Hiroshi Yamamoto,et al.  Reduction of required precision bits for back-propagation applied to pattern recognition , 1993, IEEE Trans. Neural Networks.

[24]  Nouma Izeboudjen,et al.  Digital Implementation of Artificial Neural Networks: From VHDL Description to EPGA Implementation , 1999, IWANN.

[25]  E. Fiesler,et al.  Hardware-friendly learning algorithms for neural networks: an overview , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[26]  Naleih M. Botros,et al.  Hardware implementation of an artificial neural network , 1993, IEEE International Conference on Neural Networks.

[27]  Gérard Dreyfus,et al.  Specification and implementation of a digital Hopfield-type associative memory with on-chip training , 1992, IEEE Trans. Neural Networks.

[28]  H. C. Zeidler,et al.  On-chip backpropagation training using parallel stochastic bit streams , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[29]  Bernard Girau,et al.  FPNA: Interaction Between FPGA and Neural Computation , 2000, Int. J. Neural Syst..

[30]  Marwan A. Jabri,et al.  PANNE: a parallel computing engine for connectionist simulation , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[31]  J.-L. Beuchat.  Conception d"un neuroprocesseur reconfigurable proposant des algorithmes d"apprentissage et d"élagage: une première étude , 1998 .

[32]  Paolo Ienne,et al.  MANTRA: a multi-model neural-network computer , 1992 .

[33]  Michael K. Gschwind,et al.  A Generic Building Block For Hopfield Neural Networks With On-chip Learning , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[34]  Thomas Kailath,et al.  Depth-Size Tradeoffs for Neural Computation , 1991, IEEE Trans. Computers.

[35]  Joachim K. Anlauf,et al.  Fast Digital Simulation of Spiking Neural Networks and Neuromorphic Integration with Spikelab , 1999, Int. J. Neural Syst..

[36]  Hélène Paugam-Moisy,et al.  Size of Multilayer Networks for Exact Learning: Analytic Approach , 1996, NIPS.

[37]  Francisco José Ballester-Merelo,et al.  Artificial neural network implementation on a single FPGA of a pipelined on-line backpropagation , 2000, Proceedings 13th International Symposium on System Synthesis.

[38]  John J. Paulos,et al.  The Effects of Precision Constraints in a Backpropagation Learning Network , 1990, Neural Computation.

[39]  Bernard Girau.  Digital hardware implementation of 2D compatible neural networks , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[40]  Bernard Girau.  Du parallelisme des modeles connexionnistes a leur implantation parallele , 1999 .

[41]  Y. Boniface.  A parallel simulator to build distributed neural algorithms , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[42]  George W. Irwin,et al.  Fast parallel off-line training of multilayer perceptrons , 1997, IEEE Trans. Neural Networks.

[43]  Bernard Girau.  Simplified neural architectures for symmetric boolean functions , 2000, ESANN.

[44]  David Abramson,et al.  FPGA based implementation of a Hopfield neural network for solving constraint satisfaction problems , 1998, Proceedings. 24th EUROMICRO Conference (Cat. No.98EX204).

[45]  Demessie Girma,et al.  Artificial Neural Network Implementation on a Fine-Grained FPGA , 1994, FPL.

[46]  Bertil Svensson,et al.  Using and Designing Massively Parallel Computers for Artificial Neural Neural Networks , 1992, J. Parallel Distributed Comput..

[47]  Alan Kramer.  Array-based analog computation: principles, advantages and limitations , 1996, Proceedings of Fifth International Conference on Microelectronics for Neural Networks.

[48]  Bernard Girau.  Neural networks on FPGAs: a survey , 1999 .

[49]  Hiroomi Hikawa.  Frequency-based multilayer neural network with on-chip learning and enhanced neuron characteristics , 1999, IEEE Trans. Neural Networks.

[50]  A. Petrowski.  Choosing among several parallel implementations of the backpropagation algorithm , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[51]  Christof Teuscher,et al.  A networked FPGA-based hardware implementation of a neural network application , 2000, Proceedings 2000 IEEE Symposium on Field-Programmable Custom Computing Machines (Cat. No.PR00871).

[52]  Minesh B. Amin,et al.  A Scalable Parallel Formulation of the Backpropagation Algorithm for Hypercubes and Related Architectures , 1994, IEEE Trans. Parallel Distributed Syst..

[53]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[54]  Valentina Salapura.  Neural networks using bit stream arithmetic: a space efficient implementation , 1994, Proceedings of IEEE International Symposium on Circuits and Systems - ISCAS '94.

[55]  B. Girau.  On-chip learning of FPGA-inspired neural nets , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[56]  Andrés Pérez-Uribe,et al.  FPGA Implementation of an Adaptable-Size Neural Network , 1996, ICANN.

[57]  Anton Gunzinger,et al.  Fast neural net simulation with a DSP processor array , 1995, IEEE Trans. Neural Networks.

[58]  Qinghua Zhang,et al.  Wavelet networks , 1992, IEEE Trans. Neural Networks.

[59]  S. R. Jones,et al.  Arithmetic Unit Design for Neural Accelerators: Cost Performance Issues , 1995, IEEE Trans. Computers.

[60]  Jean-Luc Gaudiot,et al.  Implementing regularly structured neural networks on the DREAM machine , 1995, IEEE Trans. Neural Networks.

[61]  Brad Hutchings,et al.  FPGA-based stochastic neural networks-implementation , 1994, Proceedings of IEEE Workshop on FPGA's for Custom Computing Machines.

[62]  Jill P. Mesirov,et al.  The backpropagation algorithm on grid and hypercube architectures , 1990, Parallel Comput..

[63]  Martin Schäfer,et al.  Simulation of spiking neural networks -- architectures and implementations , 2002, Neurocomputing.

[64]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[65]  B. Girau,et al.  Parameterized normalization: application to wavelet networks , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).