Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations

There has been much study of ASIC neurocomputers but, in comparison, relatively little for FPGA neurocomputers. Nevertheless, with current (and future) dense, high-speed FPGAs, the latter are now viable and will be more successful than the former. In this paper, we discuss a technique for low-error, high-speed implementations of the sigmoid function in such FPGAs. This function is commonly used as an activation function in artificial neural networks, but it also has applications in many other areas. Our results compare very favourably with others that have been reported in the published literature.

[1]  H.S. Abdel-Aty-Zohdy,et al.  Implementation of programmable digital sigmoid function circuit for neuro-computing , 1998, 1998 Midwest Symposium on Circuits and Systems (Cat. No. 98CB36268).

[2]  Wayne Luk,et al.  Parameterized High Throughput Function Evaluation for FPGAs , 2004, J. VLSI Signal Process..

[3]  C. Alippi,et al.  Simple approximation of sigmoidal functions: realistic design of digital neural networks capable of learning , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[4]  Stamatis Vassiliadis,et al.  Sigmoid Generators for Neural Computing Using Piecewise Approximations , 1996, IEEE Trans. Computers.

[5]  Brian Kingsbury,et al.  Spert-II: A Vector Microprocessor System , 1996, Computer.

[6]  Michael J. Flynn,et al.  Advanced Computer Arithmetic Design , 2001 .

[7]  K. M. Curtis,et al.  Piecewise linear approximation applied to nonlinear function of a neural network , 1997 .

[8]  Amos R. Omondi,et al.  Computer arithmetic systems - algorithms, architecture and implementation , 1994, Prentice Hall International series in computer science.

[9]  J. M. Tarela,et al.  Approximation of sigmoid function and the derivative for hardware implementation of artificial neurons , 2004 .

[10]  Amos Omondi.  Neurocomputers: A Dead End? , 2000, Int. J. Neural Syst..

[11]  Jagath C. Rajapakse,et al.  Neural networks in FPGAs , 2002, Proceedings of the 9th International Conference on Neural Information Processing, 2002. ICONIP '02..

[12]  Gaston H. Gonnet,et al.  On the LambertW function , 1996, Adv. Comput. Math..

[13]  Matti Tommiska,et al.  Efficient digital implementation of the sigmoid function for reprogrammable logic , 2003 .

[14]  Jean-Michel Muller,et al.  Elementary Functions: Algorithms and Implementation , 1997 .

[15]  Hon Keung Kwan.  Simple sigmoid-like activation function suitable for digital hardware implementation , 1992 .

[16]  D. J. Myers,et al.  Efficient implementation of piecewise linear activation function for digital VLSI neural networks , 1989 .

[17]  A. Fernandez,et al.  Design of a pipelined hardware architecture for real-time neural network computations , 2002, The 2002 45th Midwest Symposium on Circuits and Systems, 2002. MWSCAS-2002..

[18]  Stephen M Pizer,et al.  To compute numerically: Concepts and strategies (Little, Brown computer systems series) , 1983 .

[19]  S. R. Jones,et al.  IMPLEMENTING NONLINEAR ACTIVATION FUNCTIONS IN NEURAL NETWORK EMULATORS , 1991 .

[20]  Stamatis Vassiliadis,et al.  Elementary function generators for neural-network emulators , 2000, IEEE Trans. Neural Networks Learn. Syst..

[21]  Stephen M. Pizer,et al.  To compute numerically : concepts and strategies , 1983 .

[22]  Mariusz Bajger,et al.  Implementations of Square-Root and Exponential Functions for Large FPGAs , 2006, Asia-Pacific Computer Systems Architecture Conference.