Low-error digital hardware implementation of artificial neuron activation functions and their derivative

Abstract In this paper we propose a low-error approximation of the sigmoid function and hyperbolic tangent, which are mainly used to activate the artificial neuron, based on the piecewise linear method. Here, the hyperbolic tangent is alternatively approximated by exploiting its mathematical relationship with the sigmoid function, showing better results. Special attention has been paid to study the minimum number of precision bits to achieve the convergence of a multi-layer perceptron network in finite arithmetic machine. All the approximation results show lower mean relative and absolute error than those reported in the state-of-the-art. Finally, the sigmoid digital implementation is discussed and assessed in terms of work frequency, complexity and error in comparison with the state-of-the-art.

[1]  Yasar Becerikli,et al.  Neural Network Implementation in Hardware Using FPGAs , 2006, ICONIP.

[2]  K. M. Curtis,et al.  Piecewise linear approximation applied to nonlinear function of a neural network , 1997 .

[3]  Jack E. Volder.  The CORDIC Trigonometric Computing Technique , 1959, IRE Trans. Electron. Comput..

[4]  Huapeng Wu,et al.  High Speed VLSI Implementation of the Hyperbolic Tangent Sigmoid Function , 2008, 2008 Third International Conference on Convergence and Hybrid Information Technology.

[5]  Luca Fanucci,et al.  Low-error approximation of artificial neuron sigmoid function and its derivative , 2009 .

[6]  Sorin Draghici,et al.  On the capabilities of neural networks using limited precision weights , 2002, Neural Networks.

[7]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[8]  Matti Tommiska,et al.  Efficient digital implementation of the sigmoid function for reprogrammable logic , 2003 .

[9]  Florham Park.  Parameterized High Throughput Function Evaluation for FPGAs , .

[10]  Stamatis Vassiliadis,et al.  Sigmoid Generators for Neural Computing Using Piecewise Approximations , 1996, IEEE Trans. Computers.

[11]  P. Verhulst.  Recherches mathématiques sur la loi d’accroissement de la population , 2022, Nouveaux mémoires de l'Académie royale des sciences et belles-lettres de Bruxelles.

[12]  C. Alippi,et al.  Simple approximation of sigmoidal functions: realistic design of digital neural networks capable of learning , 1991, 1991., IEEE International Sympoisum on Circuits and Systems.

[13]  Kamel Besbes,et al.  Digital Hardware Implementation of a Neural System Used for Nonlinear Adaptive Prediction , 2006 .

[14]  Emile Fiesler,et al.  Neural Network Adaptations to Hardware Implementations , 1997 .

[15]  Mariusz Bajger,et al.  Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations , 2008, J. Signal Process. Syst..

[16]  Jenq-Neng Hwang,et al.  Finite Precision Error Analysis of Neural Network Hardware Implementations , 1993, IEEE Trans. Computers.

[17]  Jeen-Shing Wang,et al.  A digital circuit design of hyperbolic tangent sigmoid function for neural networks , 2008, 2008 IEEE International Symposium on Circuits and Systems.

[18]  Shawki Areibi,et al.  The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: A Study , 2007, IEEE Transactions on Neural Networks.

[19]  J. M. Tarela,et al.  Digital design of sigmoid approximator for artificial neural networks , 2002 .

[20]  Stamatis Vassiliadis,et al.  Elementary function generators for neural-network emulators , 2000, IEEE Trans. Neural Networks Learn. Syst..

[21]  D. J. Myers,et al.  Efficient implementation of piecewise linear activation function for digital VLSI neural networks , 1989 .

[22]  Yu Hen Hu,et al.  A Novel Implementation of CORDIC Algorithm Using Backward Angle Recoding (BAR) , 1996, IEEE Trans. Computers.

[23]  J. S. Walther,et al.  A unified algorithm for elementary functions , 1899, AFIPS '71 (Spring).

[24]  George J. Milne,et al.  Towards an FPGA based reconfigurable computing environment for neural network implementations , 1999 .