High precision FPGA implementation of neural network activation functions

The efficient implementation of artificial neural networks in FPGA boards requires tackling several issues that strongly affect the final result. One of these issues is the computation of the neuron's activation function. In this work, a detailed analysis of the FPGA implementations of the Sigmoid and Exponential functions is carried out, in a approach combining a lookup table with a linear interpolation procedure. Further, to optimize board resources utilization, a time division multiplexing of the multiplier attached to the neurons was used. The results are evaluated in terms of the absolute and relative errors obtained and also through measuring a quality factor and the resource utilization, showing a clear improvement in relationship to previously published works.

[1]  S. Himavathi,et al.  Feedforward Neural Network Implementation in FPGA Using Layer Multiplexing for Effective Resource Utilization , 2007, IEEE Transactions on Neural Networks.

[2]  Jagath C. Rajapakse,et al.  FPGA Implementations of Neural Networks , 2006 .

[3]  Pong P. Chu.  FPGA Prototyping by VHDL Examples: Xilinx Spartan-3 Version , 2008 .

[4]  K. M. Curtis,et al.  Piecewise linear approximation applied to nonlinear function of a neural network , 1997 .

[5]  Javier Echanobe,et al.  Controlled accuracy approximation of sigmoid function for efficient FPGA-based implementation of artificial neurons , 2013 .

[6]  Steve Kilts.  Control System Components , 2008 .

[7]  Seul Jung,et al.  Hardware Implementation of a Real-Time Neural Network Controller With a DSP and an FPGA for Nonlinear Systems , 2007, IEEE Transactions on Industrial Electronics.

[8]  Shawki Areibi,et al.  The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: A Study , 2007, IEEE Transactions on Neural Networks.

[9]  Wayne Luk,et al.  Floating-Point FPGA: Architecture and Modeling , 2009, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[10]  Marcus Frean,et al.  The Upstart Algorithm: A Method for Constructing and Training Feedforward Neural Networks , 1990, Neural Computation.

[11]  W. Marsden.  I and J , 2012 .

[12]  Abhisek Ukil,et al.  Development and Implementation of Parameterized FPGA-Based General Purpose Neural Networks for Online Applications , 2011, IEEE Transactions on Industrial Informatics.

[13]  K. P. Lakshmi,et al.  A Survey on FPGA based MLP Realization for On-chip Learning , 2013 .

[14]  Veljko M. Milutinovic,et al.  FPGA accelerator for floating-point matrix multiplication , 2012, IET Comput. Digit. Tech..

[15]  Jihan Zhu,et al.  FPGA Implementations of Neural Networks - A Survey of a Decade of Progress , 2003, FPL.

[16]  C. Alippi,et al.  Simple approximation of sigmoidal functions: realistic design of digital neural networks capable of learning , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[17]  D. J. Myers,et al.  Efficient implementation of piecewise linear activation function for digital VLSI neural networks , 1989 .

[18]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[19]  Matti Tommiska,et al.  Efficient digital implementation of the sigmoid function for reprogrammable logic , 2003 .

[20]  Leonardo Franco,et al.  C-Mantec: A novel constructive neural network algorithm incorporating competition between neurons , 2012, Neural Networks.

[21]  Eric Monmasson,et al.  FPGAs in Industrial Control Applications , 2011, IEEE Transactions on Industrial Informatics.

[22]  Stamatis Vassiliadis,et al.  Sigmoid Generators for Neural Computing Using Piecewise Approximations , 1996, IEEE Trans. Computers.