P-SFA: Probability based Sigmoid Function Approximation for Low-complexity Hardware Implementation

ABSTRACT A probability-based sigmoid function approximation (P-SFA), which is based on piecewise linear function and neuron's values statistical probability distribution in each layer, is proposed to lower the complexity of neural network hardware implementation with only addition circuit. The sigmoid function is divided into three fixed regions, and the number of sub-regions with different sizes in each fixed region is adapted to neuron's values distribution in each layer to reduce the error between the sigmoid function and P-SFA function. The experimental results on FPGA show that the P-SFA function is efficient in terms of power and speed, and the recognition accuracies in DNN and CNN for MNIST with P-SFA are the highest among the state-of-the-art methods, up to 97.46% and 99.02% respectively.

[1]  Sujan Kumar Gonugondla,et al.  An In-Memory VLSI Architecture for Convolutional Neural Networks , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[2]  Fernando Morgado Dias,et al.  A new solution to the hyperbolic tangent implementation in hardware: polynomial modeling of the fractional exponential part , 2013, Neural Computing and Applications.

[3]  Mariusz Bajger,et al.  Low-error, High-speed Approximation of the Sigmoid Function for Large FPGA Implementations , 2008, J. Signal Process. Syst..

[4]  Fernando Morgado Dias,et al.  A high bit resolution FPGA implementation of a FNN with a new algorithm for the activation function , 2007, Neurocomputing.

[5]  Douglas A. Reynolds,et al.  Deep Neural Network Approaches to Speaker and Language Recognition , 2015, IEEE Signal Processing Letters.

[6]  Mitra Mirhassani,et al.  Efficient VLSI Implementation of Neural Networks With Hyperbolic Tangent Activation Function , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[7]  Xiang Bai,et al.  An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Hoi-Jun Yoo,et al.  4.6 A1.93TOPS/W scalable deep learning/inference processor with tetra-parallel MIMD architecture for big-data applications , 2015, 2015 IEEE International Solid-State Circuits Conference - (ISSCC) Digest of Technical Papers.

[9]  R. Venkatesh Babu,et al.  DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations , 2015, IEEE Transactions on Image Processing.

[10]  Fevzullah Temurtas,et al.  A Study on Neural Networks Using Taylor Series Expansion of Sigmoid Activation Function , 2004, ICCSA.

[11]  Majid Ahmadi,et al.  Precise digital implementations of hyperbolic tanh and sigmoid function , 2016, 2016 50th Asilomar Conference on Signals, Systems and Computers.

[12]  Stamatis Vassiliadis,et al.  Sigmoid Generators for Neural Computing Using Piecewise Approximations , 1996, IEEE Trans. Computers.

[13]  Qi Fan,et al.  A fast BP networks with dynamic sample selection for handwritten recognition , 2018, Pattern Analysis and Applications.

[14]  Mitra Mirhassani,et al.  An Analog CVNS-Based Sigmoid Neuron for Precise Neurochips , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[15]  L. Fanucci,et al.  Low-error digital hardware implementation of artificial neuron activation functions and their derivative , 2011, Microprocess. Microsystems.

[16]  Nilay Khare,et al.  Hardware implementation of neural network with Sigmoidal activation functions using CORDIC , 2015, Microprocess. Microsystems.

[17]  Qingxiang Wu,et al.  Image super-resolution using a dilated convolutional neural network , 2018, Neurocomputing.

[18]  Raymond W. Ptucha,et al.  Intelligent character recognition using fully convolutional neural networks , 2019, Pattern Recognit..

[19]  Javier Echanobe,et al.  Controlled accuracy approximation of sigmoid function for efficient FPGA-based implementation of artificial neurons , 2013 .

[20]  Chen-Yi Lee,et al.  A Hardware-Efficient Sigmoid Function With Adjustable Precision for a Neural Network System , 2015, IEEE Transactions on Circuits and Systems II: Express Briefs.

[21]  Jih-Gau Juang,et al.  Automatic Landing Control System Design Using Adaptive Neural Network and Its Hardware Realization , 2011, IEEE Systems Journal.

[22]  Paramita Chattopadhyay,et al.  Challenges in implementation of ANN in embedded system , 2016, 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT).

[23]  Qiang Liu,et al.  Fast Neural Network Training on FPGA Using Quasi-Newton Optimization Method , 2018, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[24]  Taras Iakymchuk,et al.  A Novel Systolic Parallel Hardware Architecture for the FPGA Acceleration of Feedforward Neural Networks , 2019, IEEE Access.

[25]  Syahrulanuar Ngah,et al.  Sigmoid Function Implementation Using the Unequal Segmentation of Differential Lookup Table and Second Order Nonlinear Function , 2017 .

[26]  Shawki Areibi,et al.  The Impact of Arithmetic Representation on Implementing MLP-BP on FPGAs: A Study , 2007, IEEE Transactions on Neural Networks.