Extreme Learning Machine and Its Applications in Big Data Processing

Abstract The extreme learning machine (ELM) is widely used in batch learning, sequential learning, and incremental learning because of its fast and efficient learning speed, fast convergence, good generalization ability, and ease of implementation. With the development of the traditional ELM, lots of improved ELM algorithms have been proposed; meanwhile the scope of implementing the ELM has been further expanded from supervised learning, to semisupervised learning and unsupervised learning. However, due to its memory-residency, and high space and time complexity, the traditional ELM is not able to train big data fast and efficiently. Optimization strategies have been employed for the traditional ELM to solve this problem. In this chapter, we will first review ELM theories and some important variants, and then describe parallel ELM algorithms based on MapReduce and Spark in detail. Lastly, we show some practical applications of the ELM for big data.

[1]  W. Zhang,et al.  Fuzzy extreme learning machine for classification , 2013 .

[2]  Sanggil Kang,et al.  A fusion neural network classifier for image classification , 2009, Pattern Recognit. Lett..

[3]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[4]  Guang-Bin Huang,et al.  Learning capability and storage capacity of two-hidden-layer feedforward networks , 2003, IEEE Trans. Neural Networks.

[5]  M. Sternberg,et al.  Prediction of protein secondary structure and active sites using the alignment of homologous sequences. , 1987, Journal of molecular biology.

[6]  Weiyang Zhou,et al.  Verification of the nonparametric characteristics of backpropagation neural networks for image classification , 1999, IEEE Trans. Geosci. Remote. Sens..

[7]  Antonio J. Serrano,et al.  BELM: Bayesian Extreme Learning Machine , 2011, IEEE Transactions on Neural Networks.

[8]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[9]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[10]  Piotr Indyk,et al.  Approximate nearest neighbors: towards removing the curse of dimensionality , 1998, STOC '98.

[11]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[12]  Han Wang,et al.  Evolutionary Extreme Learning Machine and Its Application to Image Analysis , 2013, J. Signal Process. Syst..

[13]  Cheng Wu,et al.  Semi-Supervised and Unsupervised Extreme Learning Machines , 2014, IEEE Transactions on Cybernetics.

[14]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[15]  Nicole Immorlica,et al.  Locality-sensitive hashing scheme based on p-stable distributions , 2004, SCG '04.

[16]  Guang-Bin Huang,et al.  Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions , 1998, IEEE Trans. Neural Networks.

[17]  J. Manyika Big data: The next frontier for innovation, competition, and productivity , 2011 .

[18]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[19]  Wei Wu,et al.  Evolutionary Fuzzy Extreme Learning Machine for Mammographic Risk Analysis , 2011 .

[20]  C. A. Murthy,et al.  Formulation of a multivalued recognition system , 1992, IEEE Trans. Syst. Man Cybern..

[21]  Q. M. Jonathan Wu,et al.  A Combination of Positive and Negative Fuzzy Rules for Image Classification Problem , 2008, 2008 Seventh International Conference on Machine Learning and Applications.

[22]  Constantin F. Aliferis,et al.  A comprehensive evaluation of multicategory classification methods for microarray gene expression cancer diagnosis , 2004, Bioinform..

[23]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[24]  Tao Li,et al.  A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression , 2004, Bioinform..

[25]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[26]  Halbert White,et al.  Artificial Neural Networks: Approximation and Learning Theory , 1992 .

[27]  Sankar K. Pal,et al.  Linguistic recognition system based on approximate reasoning , 1992, Inf. Sci..

[28]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[29]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[30]  Dong Sun Park,et al.  Online sequential extreme learning machine with forgetting mechanism , 2012, Neurocomputing.

[31]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[32]  Rajkumar Buyya,et al.  Big Data computing and clouds: Trends and future directions , 2013, J. Parallel Distributed Comput..

[33]  Guang-Bin Huang,et al.  Trends in extreme learning machines: A review , 2015, Neural Networks.

[34]  H. D. Block The perceptron: a model for brain functioning. I , 1962 .

[35]  Sundaram Suresh,et al.  Fast learning Circular Complex-valued Extreme Learning Machine (CC-ELM) for real-valued classification problems , 2012, Inf. Sci..

[36]  Hongming Zhou,et al.  Optimization method based extreme learning machine for classification , 2010, Neurocomputing.

[37]  J. Garnier,et al.  Analysis of the accuracy and implications of simple methods for predicting the secondary structure of globular proteins. , 1978, Journal of molecular biology.

[38]  Dong Ling Tong,et al.  Genetic Algorithm-Neural Network (GANN): a study of neural network activation functions and depth of genetic algorithm search applied to feature selection , 2010, Int. J. Mach. Learn. Cybern..

[39]  Hongming Zhou,et al.  Extreme Learning Machines [Trends & Controversies] , 2013 .

[40]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[41]  Jae Won Lee,et al.  An extensive comparison of recent classification tools applied to microarray data , 2004, Comput. Stat. Data Anal..

[42]  S. Brunak,et al.  Predicting subcellular localization of proteins based on their N-terminal amino acid sequence. , 2000, Journal of molecular biology.

[43]  Albert Y. Zomaya,et al.  Recent advances in autonomic provisioning of big data applications on clouds , 2015, IEEE Trans. Cloud Comput..

[44]  Chi-Man Vong,et al.  Sparse Bayesian Extreme Learning Machine for Multi-classification , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[45]  Yaonan Wang,et al.  Bidirectional Extreme Learning Machine for Regression Problem and Its Learning Effectiveness , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[46]  Taoying Li,et al.  Opportunities of innovation under challenges of big data , 2013, 2013 10th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD).

[47]  Hisao Ishibuchi,et al.  Rule weight specification in fuzzy rule-based classification systems , 2005, IEEE Transactions on Fuzzy Systems.

[48]  Feilong Cao,et al.  A study on effectiveness of extreme learning machine , 2011, Neurocomputing.

[49]  Dipankar Das,et al.  Enhanced SenticNet with Affective Labels for Concept-Based Opinion Mining , 2013, IEEE Intelligent Systems.

[50]  Mohammad Reza Daliri,et al.  A Hybrid Automatic System for the Diagnosis of Lung Cancer Based on Genetic Algorithm and Fuzzy Extreme Learning Machines , 2012, Journal of Medical Systems.

[51]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[52]  M. Ringnér,et al.  Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks , 2001, Nature Medicine.

[53]  Hisao Ishibuchi,et al.  A weighted fuzzy classifier and its application to image processing tasks , 2007, Fuzzy Sets Syst..

[54]  Korris Fu-Lai Chung,et al.  Positive and negative fuzzy rule system, extreme learning machine and image classification , 2011, Int. J. Mach. Learn. Cybern..

[55]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.

[56]  P. Saratchandran,et al.  Multicategory Classification Using An Extreme Learning Machine for Microarray Gene Expression Cancer Diagnosis , 2007, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[57]  Zhen Lin Bioinformatics Basics: Applications in Biological Science and Medicine.: Edited by Lukas K. Buehler and Hooman H. Rashidi , 2008, Briefings Bioinform..

[58]  Burkhard Rost,et al.  Rising Accuracy of Protein Secondary Structure Prediction , 2003 .

[59]  Kun-Shan Chen,et al.  A fuzzy neural network to SAR image classification , 1998, IEEE Trans. Geosci. Remote. Sens..

[60]  Kenli Li,et al.  Performance Analysis and Optimization for SpMV on GPU Using Probabilistic Modeling , 2015, IEEE Transactions on Parallel and Distributed Systems.

[61]  Christos Doulkeridis,et al.  A survey of large-scale analytical query processing in MapReduce , 2013, The VLDB Journal.

[62]  Dongmei Liu,et al.  A united classification system of X-ray image based on fuzzy rule and neural networks , 2008, 2008 3rd International Conference on Intelligent System and Knowledge Engineering.

[63]  L. K. Buehler,et al.  Bioinformatics Basics: Applications in Biological Science and Medicine , 1999 .

[64]  Robert K. L. Gay,et al.  Error Minimized Extreme Learning Machine With Growth of Hidden Nodes and Incremental Learning , 2009, IEEE Transactions on Neural Networks.

[65]  Kenli Li,et al.  Performance Optimization Using Partitioned SpMV on GPUs and Multicore CPUs , 2015, IEEE Transactions on Computers.

[66]  Xuelong Li,et al.  Single-image super-resolution via local learning , 2011, Int. J. Mach. Learn. Cybern..

[67]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[68]  H. D. Block,et al.  Analysis of a Four-Layer Series-Coupled Perceptron. II , 1962 .

[69]  Siegfried J. Pöppl,et al.  The 'subsequent artificial neural network' (SANN) approach might bring more classificatory power to ANN-based DNA microarray analyses , 2004, Bioinform..

[70]  Joseph K. Liu,et al.  Toward efficient and privacy-preserving computing in big data era , 2014, IEEE Network.

[71]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[72]  Yiqiang Chen,et al.  Weighted extreme learning machine for imbalance learning , 2013, Neurocomputing.

[73]  Dianhui Wang,et al.  Extreme learning machines: a survey , 2011, Int. J. Mach. Learn. Cybern..

[74]  D. Lowe,et al.  Adaptive radial basis function nonlinearities, and the problem of generalisation , 1989 .

[75]  Danwei Wang,et al.  Sparse Extreme Learning Machine for Classification , 2014, IEEE Transactions on Cybernetics.

[76]  Guang-Bin Huang,et al.  Extreme learning machine: a new learning scheme of feedforward neural networks , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[77]  Fuzhen Zhuang,et al.  Parallel extreme learning machine for regression based on MapReduce , 2013, Neurocomputing.

[78]  D. Mount Bioinformatics: Sequence and Genome Analysis , 2001 .

[79]  Yi Zhao,et al.  A protein secondary structure prediction framework based on the Extreme Learning Machine , 2008, Neurocomputing.

[80]  Zhiqiong Wang,et al.  ELM ∗ : distributed extreme learning machine with MapReduce , 2013, World Wide Web.

[81]  Narasimhan Sundararajan,et al.  Online Sequential Fuzzy Extreme Learning Machine for Function Approximation and Classification Problems , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[82]  Vipin Kumar,et al.  Trends in big data analytics , 2014, J. Parallel Distributed Comput..

[83]  Ronei Marcos de Moraes,et al.  Fuzzy expert systems architecture for image classification using mathematical morphology operators , 2002, Inf. Sci..

[84]  Yu Liu,et al.  Parallel online sequential extreme learning machine based on MapReduce , 2015, Neurocomputing.