Analog programmable neuron and case study on VLSI implementation of Multi-Layer Perceptron (MLP)

Abstract In this work, a programmable neuron is proposed to approximate the following activation functions: sigmoid, hyperbolic tangent and linear. In other words, the neural network designer can use several control bits to choose the type of activation function without any physical change. The proposed neuron which is simulated in a 0.18 μm CMOS technology, shows a good approximation with maximum error from the ideal hyperbolic tangent function and the ideal sigmoid function by 29.33% and 7.4%, respectively. In order to evaluate the functionality of the neuron, it is applied in two Multi-Layer Perceptron (MLP) neural networks. The first one which is trained to implement XOR gate, is capable of processing signals in the frequency range from 2.5 mHz to 50 MHz. The accuracy of the network is more than 99.9%. The second one is a pattern recognition neural network. Comparison with previous work reveals that there is 48% decrease in the network's power consumption. Moreover, the proposed neuron has been applied in the fully connected (FC) layers of a convolutional neural network (CNN) and an experiment has been conducted on a benchmark dataset MNIST.

[1]  Hoi-Jun Yoo,et al.  An Energy-Efficient Embedded Deep Neural Network Processor for High Speed Visual Attention in Mobile Vision Recognition SoC , 2016, IEEE Journal of Solid-State Circuits.

[2]  Rob Fergus,et al.  Stochastic Pooling for Regularization of Deep Convolutional Neural Networks , 2013, ICLR.

[3]  Leon O. Chua,et al.  Memristor Bridge Synapse-Based Neural Network and Its Learning , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Zhongsheng Hou,et al.  Data-Driven Modeling for UGI Gasification Processes via an Enhanced Genetic BP Neural Network With Link Switches , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[5]  Mamoun F. Abdel-Hafez,et al.  Non-Linear Autoregressive Delay-Dependent INS/GPS Navigation System Using Neural Networks , 2017, IEEE Sensors Journal.

[6]  Majid Ahmadi,et al.  Analog Implementation of a Novel Resistive-Type Sigmoidal Neuron , 2012, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[7]  Mohammad Teshnehlab,et al.  Feature Extraction and Classification of EEG Signals Using Wavelet Transform, SVM and Artificial Neural Networks for Brain Computer Interfaces , 2009, 2009 International Joint Conference on Bioinformatics, Systems Biology and Intelligent Computing.

[8]  Tulay Yildirim,et al.  Low voltage low power neuron circuit design based on subthreshold FGMOS transistors and XOR implementation , 2010, 2010 XIth International Workshop on Symbolic and Numerical Methods, Modeling and Applications to Circuit Design (SM2ACD).

[9]  Robert J. Schalkoff,et al.  Artificial neural networks , 1997 .

[10]  Abbes Amira,et al.  MLP Neural Network Based Gas Classification System on Zynq SoC , 2016, IEEE Access.

[11]  Nima Tajbakhsh,et al.  Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? , 2016, IEEE Transactions on Medical Imaging.

[12]  Hossein Shamsi,et al.  Extracting trade-off boundaries of CMOS two-stage op-amp using particle swarm optimization , 2009, 2009 International Symposium on Signals, Circuits and Systems.

[13]  Mohammad Teshnehlab,et al.  Feature selection and classification of breast MRI lesions based on multi classifier , 2011, 2011 International Symposium on Artificial Intelligence and Signal Processing (AISP).

[14]  Jiwen Lu,et al.  PCANet: A Simple Deep Learning Baseline for Image Classification? , 2014, IEEE Transactions on Image Processing.

[15]  Jia Li,et al.  Hyperbolic linear units for deep convolutional neural networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[16]  Jaime Lloret,et al.  Short-Term Load Forecasting for Microgrids Based on Artificial Neural Networks , 2013 .

[17]  Chi-Man Vong,et al.  Sparse Bayesian Extreme Learning Machine for Multi-classification , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[18]  Hossein Shamsi,et al.  Multi-objective design automation of the folded-cascode OP-AMP using NSGA-II Strategy , 2009, 2009 International Symposium on Signals, Circuits and Systems.

[19]  R. Ransohoff,et al.  The myeloid cells of the central nervous system parenchyma , 2010, Nature.

[20]  Mitra Mirhassani,et al.  An Analog CVNS-Based Sigmoid Neuron for Precise Neurochips , 2017, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[21]  Huibing Wang,et al.  Deep CNNs With Spatially Weighted Pooling for Fine-Grained Car Recognition , 2017, IEEE Transactions on Intelligent Transportation Systems.

[22]  Selahattin Sayil,et al.  Soft Error Mitigation Using Transmission Gate With Varying Gate and Body Bias , 2017, IEEE Design & Test.

[23]  Kofi Odame,et al.  Deep Neural Networks for Identifying Cough Sounds , 2016, IEEE Transactions on Biomedical Circuits and Systems.

[24]  Vipan Kakkar,et al.  Comparative Study on Analog and Digital Neural Networks , 2009 .

[25]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[26]  Angshul Majumdar,et al.  Discriminative Robust Deep Dictionary Learning for Hyperspectral Image Classification , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[27]  Neeta Pandey,et al.  Implementation of asynchronous pipeline using Transmission Gate logic , 2016, 2016 International Conference on Computational Techniques in Information and Communication Technologies (ICCTICT).

[28]  Majid Ahmadi,et al.  A unified synapse-neuron building block for hybrid VLSI neural networks , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[29]  Sattar Mirzakuchaki,et al.  A novel fast, low-power and high-performance XOR-XNOR cell , 2016, 2016 IEEE International Symposium on Circuits and Systems (ISCAS).

[30]  Laurene V. Fausett,et al.  Fundamentals Of Neural Networks , 1994 .

[31]  J. M. Tarela,et al.  Approximation of sigmoid function and the derivative for hardware implementation of artificial neurons , 2004 .

[32]  Seul Jung,et al.  Evaluation of embedded RBF neural chip with back-propagation algorithm for pattern recognition tasks , 2008, 2008 6th IEEE International Conference on Industrial Informatics.

[33]  A. Guyton,et al.  Textbook of Medical Physiology , 1961 .

[34]  Gang Pan,et al.  SparseConnect: regularising CNNs on fully connected layers , 2017 .

[35]  Huanhuan Chen,et al.  Evolving Least Squares Support Vector Machines for Stock Market Trend Mining , 2009, IEEE Trans. Evol. Comput..

[36]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[37]  R. J. Sassi,et al.  Study on Daily Demand Forecasting Orders using Artificial Neural Network , 2016, IEEE Latin America Transactions.

[38]  Nitin Mantri,et al.  BP-ANN for Fitting the Temperature-Germination Model and Its Application in Predicting Sowing Time and Region for Bermudagrass , 2013, PloS one.

[39]  Muhammad Yahuza.  UTILIZING ARTIFICIAL NEURAL NETWORK FOR PREDICTION IN THE NIGERIAN STOCK MARKET PRICE INDEX , 2011 .

[40]  Majid Ahmadi,et al.  Hyperbolic tangent passive resistive-type neuron , 2015, 2015 IEEE International Symposium on Circuits and Systems (ISCAS).

[41]  Yu Tsao,et al.  Personalizing Recurrent-Neural-Network-Based Language Model by Social Network , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[42]  Yu Tsao,et al.  S1 and S2 Heart Sound Recognition Using Deep Neural Networks , 2017, IEEE Trans. Biomed. Eng..

[43]  Takashi Maruyama,et al.  Determination of GPS receiver differential biases by neural network parameter estimation method , 2005 .

[44]  Seong-Ook Park,et al.  Drone Classification Using Convolutional Neural Networks With Merged Doppler Images , 2017, IEEE Geoscience and Remote Sensing Letters.

[45]  Oguz Akbilgic,et al.  A novel Hybrid RBF Neural Networks model as a forecaster , 2013, Statistics and Computing.

[46]  Mohamed A. Elgamel,et al.  Design methodologies for high-performance noise-tolerant XOR-XNOR circuits , 2006, IEEE Transactions on Circuits and Systems I: Regular Papers.

[47]  Yu Zhou,et al.  Polarimetric SAR Image Classification Using Deep Convolutional Neural Networks , 2016, IEEE Geoscience and Remote Sensing Letters.

[48]  Chao-Ming Huang,et al.  One-day-ahead hourly forecasting for photovoltaic power generation using an intelligent method with weather-based forecasting models , 2015 .

[49]  S. L. Pinjare,et al.  Design and Analog VLSI Implementation of Neural Network Architecture for Signal Processing , 2009 .

[50]  Tomoo Ushio,et al.  A Neural-Network-Based Beamformer for Phased Array Weather Radar , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[51]  Yannis Tsividis,et al.  A reconfigurable VLSI neural network , 1992 .

[52]  Chao Li,et al.  A Convolutional Neural Network Model for Online Medical Guidance , 2016, IEEE Access.

[53]  Hossein Shamsi,et al.  Resilient design of current steering DACs using a transistor level approach , 2017 .

[54]  Hossein Shamsi,et al.  Yield‐aware sizing of pipeline ADC using a multiple‐objective evolutionary algorithm , 2017, Int. J. Circuit Theory Appl..

[55]  Laurent Gatet,et al.  Comparison Between Analog and Digital Neural Network Implementations for Range-Finding Applications , 2009, IEEE Transactions on Neural Networks.

[56]  Adem Acır.  Application of artificial neural network to exergy performance analysis of coal fired thermal power plant , 2013 .