Mixture of ELM based experts with trainable gating network

Mixture of experts method is a neural network based ensemble learning that has great ability to improve the overall classification accuracy. This method is based on the divide and conquer principle, in which the problem space is divided between several experts by supervisition of gating network. In this paper, we propose an ensemble learning method based on mixture of experts which is named mixture of ELM based experts with trainable gating network (MEETG) to improve the computing cost and to speed up the learning process of ME. The structure of ME consists of multi layer perceptrons (MLPs) as base experts and gating network, in which gradient-based learning algorithm is applied for training the MLPs which is an iterative and time consuming process. In order to overcome on these problems, we use the advantages of extreme learning machine (ELM) for designing the structure of ME. ELM as a learning algorithm for single hidden-layer feed forward neural networks provides much faster learning process and better generalization ability in comparision with some other traditional learning algorithms. Also, in the proposed method a trainable gating network is applied to aggregate the outputs of the experts dynamically according to the input sample. Our experimental results and statistical analysis on 11 benchmark datasets confirm that MEETG has an acceptable performance in classification problems. Furthermore, our experimental results show that the proposed approach outperforms the original ELM on prediction stability and classification accuracy.

[1]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[2]  Sung-Bae Cho,et al.  Activity recognition with android phone using mixture-of-experts co-trained with labeled and unlabeled data , 2014, Neurocomputing.

[3]  Giuliano Armano,et al.  An Improved Mixture of Experts Model: Divide and Conquer Using Random Prototypes , 2011, Ensembles in Machine Learning Applications.

[4]  Reza Ebrahimpour,et al.  View-independent face recognition with Mixture of Experts , 2008, Neurocomputing.

[5]  Malcolm I. Heywood,et al.  Input partitioning to mixture of experts , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[6]  Alvaro Soto,et al.  Embedded local feature selection within mixture of experts , 2014, Inf. Sci..

[7]  Mehdi Ghatee,et al.  Root-quatric mixture of experts for complex classification problems , 2016, Expert Syst. Appl..

[8]  Jun-Hai Zhai,et al.  Ensemble dropout extreme learning machine via fuzzy integral for data classification , 2018, Neurocomputing.

[9]  John A. Mills,et al.  A mixture of experts committee machine to design compensators for intensity modulated radiation therapy , 2006, Pattern Recognit..

[10]  Qinghua Hu,et al.  Dynamic classifier ensemble using classification confidence , 2013, Neurocomputing.

[11]  Reza Ebrahimpour,et al.  Mixture of feature specified experts , 2014, Inf. Fusion.

[12]  Qihui Wu,et al.  A survey of machine learning for big data processing , 2016, EURASIP Journal on Advances in Signal Processing.

[13]  Zhi-Hua Zhou,et al.  Ensembling local learners ThroughMultimodal perturbation , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[14]  Reza Ebrahimpour,et al.  Mixture of Experts for Persian Handwritten Word Recognition , 2011 .

[15]  Hedieh Sajedi,et al.  Convolution neural network joint with mixture of extreme learning machines for feature extraction and classification of accident images , 2019, Journal of Real-Time Image Processing.

[16]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[17]  Shifei Ding,et al.  Extreme learning machine and its applications , 2013, Neural Computing and Applications.

[18]  Dae-Jong Lee,et al.  Parameter Optimization of Extreme Learning Machine Using Bacterial Foraging Algorithm , 2007 .

[19]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[20]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[21]  Mehdi Ghatee,et al.  A regularized root-quartic mixture of experts for complex classification problems , 2016, Knowl. Based Syst..

[22]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[23]  Jörg Conradt,et al.  A Mixture-of-Experts Model for Vehicle Prediction Using an Online Learning Approach , 2019, ICANN.

[24]  Iman Nekooeimehr,et al.  Adaptive semi-unsupervised weighted oversampling (A-SUWO) for imbalanced datasets , 2016, Expert Syst. Appl..

[25]  Sotiris B. Kotsiantis,et al.  Combining bagging, boosting, rotation forest and random subspace methods , 2011, Artificial Intelligence Review.

[26]  XuanLong Nguyen,et al.  Stochastic gradient based extreme learning machines for stable online learning of advanced combustion engines , 2016, Neurocomputing.

[27]  Reza Ebrahimpour,et al.  Incorporation of a Regularization Term to Control Negative Correlation in Mixture of Experts , 2012, Neural Processing Letters.

[28]  Davar Giveki,et al.  Extended Mixture of MLP Experts by Hybrid of Conjugate Gradient Method and Modified Cuckoo Search , 2012, ArXiv.

[29]  M. Bialko,et al.  Training of artificial neural networks using differential evolution algorithm , 2008, 2008 Conference on Human System Interactions.

[30]  Haigang Zhang,et al.  A Novel Improved ELM Algorithm for a Real Industrial Application , 2014 .

[31]  Reza Ebrahimpour,et al.  Mixture of experts: a literature survey , 2014, Artificial Intelligence Review.

[32]  Samuel Lukas,et al.  Backpropagation and Levenberg-Marquardt Algorithm for Training Finite Element Neural Network , 2012, 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation.

[33]  Reza Ebrahimpour,et al.  Improving combination method of NCL experts using gating network , 2011, Neural Computing and Applications.

[34]  Huchuan Lu,et al.  Saliency detection via extreme learning machine , 2016, Neurocomputing.

[35]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).