Comparative Study of Convolution Neural Network’s Relu and Leaky-Relu Activation Functions

Convolutional neural networks refer to a collection of feed-forward artificial neural networks. These networks have been implemented successfully on visual imagery. It uses a variety of perceptrons. These perceptrons are multilayered, that need very little preprocessing. Shift invariant or space invariant NN are alias for CNN, because of their architecture which is based on shared weights. It is also established on translation invariance features. In this paper, we have used rectified linear unit (Relu) and Leaky-Relu activation for inner CNN layer and softmax activation function for output layer to analyze its effect on MNIST dataset.

[1]  Nasrollah Moghaddam Charkari,et al.  Survey on deep learning methods in human action recognition , 2017, IET Comput. Vis..

[2]  Md. Zakirul Alam Bhuiyan,et al.  A Survey on Deep Learning in Big Data , 2017, 22017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC).

[3]  Takumi Ichimura,et al.  An adaptive learning method of Deep Belief Network by layer generation algorithm , 2016, 2016 IEEE Region 10 Conference (TENCON).

[4]  Luc Van Gool,et al.  Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection , 2016, ArXiv.

[5]  Wei Xiong,et al.  Stacked Convolutional Denoising Auto-Encoders for Feature Representation , 2017, IEEE Transactions on Cybernetics.

[6]  Tianqi Chen,et al.  Empirical Evaluation of Rectified Activations in Convolutional Network , 2015, ArXiv.

[7]  Masaki Aono,et al.  Open world plant image identification based on convolutional neural network , 2016, 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA).

[8]  Kanad K. Biswas,et al.  Recurrent Neural Network Based Action Recognition from 3D Skeleton Data , 2017, 2017 13th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS).

[9]  Pius Adewale Owolawi,et al.  A survey on vehicle detection based on convolution neural networks , 2017, 2017 3rd IEEE International Conference on Computer and Communications (ICCC).