Self-Adaptive Hybrid Extreme Learning Machine for Heterogeneous Neural Networks

This paper presents a hybrid algorithm for the creation of heterogeneous single layer neural networks (SLNNs). The proposed self-adaptive heterogeneous hybrid extreme learning machine (SA-He-HyELM) trains a series of SLNNs with different neuron types in the hidden layer utilizing the extreme learning machine (ELM) algorithm. These networks are evolved into heterogeneous networks (networks having different combinations of hidden neurons) with the help of a modified genetic algorithm (GA). The algorithm is able to handle two architecturally different neuron types: traditional low order (linear) units and higher order units with different transfer functions. The GA is fully self-adaptive and uses one novel hybrid crossover operator along with a self-adaptive mutation operator in order to retain ELM’s simplicity and minimize the number of parameters need tuning. The experimental part of the current paper involves testing SA-He-HyELM with traditional ELM and other three ELM-based methods. The experimental part utilized a series of regression and classification experiments on relatively large datasets. In all cases the proposed method managed to get lower MSE or higher classification accuracy when compared to the aforementioned methods.

[1]  Bin Li,et al.  The extreme learning machine learning algorithm with tunable activation function , 2012, Neural Computing and Applications.

[2]  Padmavathi Kora,et al.  Crossover Operators in Genetic Algorithms: A Review , 2017 .

[3]  Sam Kwong,et al.  EVOLVING EXTREME LEARNING MACHINE PARADIGM WITH ADAPTIVE OPERATOR SELECTION AND PARAMETER CONTROL , 2013 .

[4]  Luis M. Candanedo,et al.  Data driven prediction models of energy use of appliances in a low-energy house , 2017 .

[5]  Pınar Tüfekci,et al.  Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods , 2014 .

[6]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[7]  T. John Stonham,et al.  Partially pre-calculated weights for the backpropagation learning regime and high accuracy function mapping using continuous input RAM-based sigma-pi nets , 2000, Neural Networks.

[8]  Joshua D. Knowles,et al.  Fifty years of pulsar candidate selection: from simple filters to a new principled real-time classification approach , 2016, Monthly Notices of the Royal Astronomical Society.

[9]  T. John Stonham,et al.  Adaptive critic for sigma-pi networks , 1996, Neural Networks.

[10]  A. Kai Qin,et al.  Evolutionary extreme learning machine , 2005, Pattern Recognit..

[11]  Alexandros T. Tzallas,et al.  Hybrid extreme learning machine approach for homogeneous neural networks , 2018, Neurocomputing.

[12]  Narasimhan Sundararajan,et al.  On-Line Sequential Extreme Learning Machine , 2005, Computational Intelligence.

[13]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[14]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[15]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[16]  Kevin N. Gurney.  Learning in networks of structured hypercubes , 1989 .

[17]  De-Shuang Huang,et al.  Improved extreme learning machine for function approximation by encoding a priori information , 2006, Neurocomputing.

[18]  E. Glavas,et al.  Neural network construction using grammatical evolution , 2005, Proceedings of the Fifth IEEE International Symposium on Signal Processing and Information Technology, 2005..

[19]  Patrick van der Smagt,et al.  Introduction to neural networks , 1995, The Lancet.

[20]  Kevin N. Gurney,et al.  Training nets of hardware realizable sigma-pi units , 1992, Neural Networks.

[21]  R. Storn,et al.  On the usage of differential evolution for function optimization , 1996, Proceedings of North American Fuzzy Information Processing.

[22]  Yonggwan Won,et al.  A Robust Online Sequential Extreme Learning Machine , 2007, ISNN.

[23]  E. Massera,et al.  On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario , 2008 .

[24]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[25]  Danilo Comminiello,et al.  Online Sequential Extreme Learning Machine With Kernels , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[26]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.

[27]  Ian W. B. Grant.  Recursive Least Squares , 1987 .

[28]  Sam Kwong,et al.  Genetic Algorithms : Concepts and Designs , 1998 .

[29]  Richard Stuart Neville,et al.  Transformations of sigma-pi nets: obtaining reflected functions by reflecting weight matrices , 2002, Neural Networks.

[30]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[31]  이찬영,et al.  Recursive Least Squares 방식의 이송계모델 파라미터 식별 , 2016 .

[32]  David Zhang,et al.  Evolutionary Cost-Sensitive Extreme Learning Machine , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[33]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[34]  B. Johnson,et al.  Integrating OpenStreetMap crowdsourced data and Landsat time-series imagery for rapid land use/land cover (LULC) mapping: Case study of the Laguna de Bay area of the Philippines , 2016 .

[35]  Tapobrata Panda,et al.  Statistical Optimization of Biological Systems , 2015 .

[36]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[37]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[38]  Guang-Bin Huang,et al.  An Insight into Extreme Learning Machines: Random Neurons, Random Features and Kernels , 2014, Cognitive Computation.

[39]  Alexandros T. Tzallas,et al.  Hybrid extreme learning machine approach for heterogeneous neural networks , 2018, Neurocomputing.

[40]  Timo Similä,et al.  Multiresponse Sparse Regression with Application to Multidimensional Scaling , 2005, ICANN.