Extreme Learning Machine with Fuzzy Activation Function

Recently, an efficient learning algorithm called extreme learning machine (ELM) has been proposed for single-hidden layer feed forward neural networks (SLFNs). Unlike the traditional gradient-descent based learning algorithms which determine network weights by iterative processes, the ELM algorithm analytically determines the output weights with random choice of input weights and hidden layer biases. This algorithm can achieve good performance with very high learning speed. In this paper, we propose a novel fuzzy-based activation function for SLFNs trained by ELM algorithm. This is a simple sigmoid-like nonlinear activation function and more suitable for hardware implementation. The experimental results for real applications show that this activation function offers good performance which is compatible to the sigmoidal activation function.

[1]  Yoshifusa Ito,et al.  Approximation of continuous functions on Rd by linear combinations of shifted rotations of a sigmoid function with and without scaling , 1992, Neural Networks.

[2]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[3]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[4]  H. A. Babri,et al.  General approximation theorem on feedforward networks , 1997, Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat..

[5]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[6]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[7]  Yoshifusa Ito,et al.  Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory , 1991, Neural Networks.

[8]  M. Viberg,et al.  Adaptive neural nets filter using a recursive Levenberg-Marquardt search direction , 1998, Conference Record of Thirty-Second Asilomar Conference on Signals, Systems and Computers (Cat. No.98CH36284).

[9]  Christopher J. Merz,et al.  UCI Repository of Machine Learning Databases , 1996 .

[10]  Tommy W. S. Chow,et al.  Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients , 2001, IEEE Trans. Neural Networks.

[11]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[12]  Bernard Widrow,et al.  Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[13]  Anastasios N. Venetsanopoulos,et al.  Artificial neural networks - learning algorithms, performance evaluation, and applications , 1992, The Kluwer international series in engineering and computer science.

[14]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[15]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[16]  D. Serre.  Matrices: Theory and Applications , 2002 .

[17]  Hong Chen,et al.  Approximation capability in C(R¯n) by multilayer feedforward networks and related problems , 1995, IEEE Trans. Neural Networks.

[18]  Antonio J. Serrano,et al.  A low-complexity fuzzy activation function for artificial neural networks , 2003, IEEE Trans. Neural Networks.

[19]  Tahseen Ahmed Jilani,et al.  A Comparison of First and Second Order Training Algorithms for Artificial Neural Networks , 2007, International Conference on Computational Intelligence.