A Connection between Extreme Learning Machine and Neural Network Kernel

We study a connection between extreme learning machine (ELM) and neural network kernel (NNK). NNK is derived from a neural network with an infinite number of hidden units. We interpret ELM as an approximation to this infinite network. We show that ELM and NNK can, to certain extent, replace each other. ELM can be used to form a kernel, and NNK can be decomposed into feature vectors to be used in the hidden layer of ELM. The connection reveals possible importance of weight variance as a parameter of ELM. Based on our experiments, we recommend that model selection on ELM should consider not only the number of hidden units, as is the current practice, but also the variance of weights. We also study the interaction of variance and the number of hidden units, and discuss some properties of ELM, that may have been too strongly interpreted previously.

[1]  A. Kai Qin,et al.  Evolutionary extreme learning machine , 2005, Pattern Recognit..

[2]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[3]  Bernhard Schölkopf,et al.  A Primer on Kernel Methods , 2004 .

[4]  A. Householder,et al.  Discussion of a set of points in terms of their mutual distances , 1938 .

[5]  Bernhard Schölkopf,et al.  Kernel Methods in Computational Biology , 2005 .

[6]  P. McCullagh,et al.  Generalized Linear Models, 2nd Edn. , 1990 .

[7]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[8]  Steve R. Gunn,et al.  Result Analysis of the NIPS 2003 Feature Selection Challenge , 2004, NIPS.

[9]  R. Penrose.  A Generalized inverse for matrices , 1955 .

[10]  P. McCullagh,et al.  An outline of generalized linear models , 1983 .

[11]  Gene H. Golub,et al.  Matrix computations (3rd ed.) , 1996 .

[12]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[13]  Benoît Frénay,et al.  Using SVMs with randomised feature spaces: an extreme learning approach , 2010, ESANN.

[14]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[15]  Gene H. Golub,et al.  Matrix computations , 1983 .

[16]  Christopher K. I. Williams.  Computation with Infinite Neural Networks , 1998, Neural Computation.

[17]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[18]  Amaury Lendasse,et al.  Interpreting Extreme Learning Machine as an Approximation to an Infinite Neural Network , 2010, KDIR.

[19]  Lawrence K. Saul,et al.  Kernel Methods for Deep Learning , 2009, NIPS.

[20]  Amaury Lendasse,et al.  OP-ELM: Optimally Pruned Extreme Learning Machine , 2010, IEEE Transactions on Neural Networks.