Ordinal extreme learning machine

Recently, a new fast learning algorithm called Extreme Learning Machine (ELM) has been developed for Single-Hidden Layer Feedforward Networks (SLFNs) in G.-B. Huang, Q.-Y. Zhu and C.-K. Siew ''[Extreme learning machine: theory and applications,'' Neurocomputing 70 (2006) 489-501]. And, ELM has been successfully applied to many classification and regression problems. In this paper, the ELM algorithm is further studied for ordinal regression problems (named ORELM). We firstly proposed an encoding-based framework for ordinal regression which includes three encoding schemes: single multi-output classifier, multiple binary-classifications with one-against-all (OAA) decomposition method and one-against-one (OAO) method. Then, the SLFN was redesigned for ordinal regression problems based on the proposed framework and the algorithms are trained by the extreme learning machine in which input weights are assigned randomly and output weights can be decided analytically. Lastly widely experiments on three kinds of datasets were carried to test the proposed algorithm. The comparative results with such traditional methods as Gaussian Process for Ordinal Regression (ORGP) and Support Vector for Ordinal Regression (ORSVM) show that ORELM can obtain extremely rapid training speed and good generalization ability. Especially when the data set's scalability increases, the advantage of ORELM will become more apparent. Additionally, ORELM has the following advantages, including the capabilities of learning in both online and batch modes and handling non-linear data.

[1]  Koby Crammer,et al.  Pranking with Ranking , 2001, NIPS.

[2]  Dan Roth,et al.  Learnability of Bipartite Ranking Functions , 2005, COLT.

[3]  Ulrich Paquet,et al.  Bayesian Hierarchical Ordinal Regression , 2005, ICANN.

[4]  Guang-Bin Huang,et al.  Convex incremental extreme learning machine , 2007, Neurocomputing.

[5]  Chee Kheong Siew,et al.  Real-time learning capability of neural networks , 2006, IEEE Trans. Neural Networks.

[6]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[7]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[8]  Narasimhan Sundararajan,et al.  A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks , 2006, IEEE Transactions on Neural Networks.

[9]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[10]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[11]  Narasimhan Sundararajan,et al.  Classification of Mental Tasks from Eeg Signals Using Extreme Learning Machine , 2006, Int. J. Neural Syst..

[12]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[13]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[14]  Yew-Soon Ong,et al.  Extreme learning machine for multi-categories classification applications , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[15]  Thomas Hofmann,et al.  Unifying collaborative and content-based filtering , 2004, ICML.

[16]  Wei Chu,et al.  Support Vector Ordinal Regression , 2007, Neural Computation.

[17]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[18]  Yoram Singer,et al.  Log-Linear Models for Label Ranking , 2003, NIPS.

[19]  Lei Chen,et al.  Enhanced random search based incremental extreme learning machine , 2008, Neurocomputing.

[20]  Dan Roth,et al.  Constraint Classification: A New Approach to Multiclass Classification , 2002, ALT.

[21]  D. Serre.  Matrices: Theory and Applications , 2002 .

[22]  Yuan Lan,et al.  Ensemble of online sequential extreme learning machine , 2009, Neurocomputing.

[23]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[24]  Gerhard Widmer,et al.  Prediction of Ordinal Classes Using Regression Trees , 2001, Fundam. Informaticae.

[25]  Amnon Shashua,et al.  Ranking with Large Margin Principle: Two Approaches , 2002, NIPS.

[26]  Chee Kheong Siew,et al.  Incremental extreme learning machine with fully complex hidden nodes , 2008, Neurocomputing.

[27]  Xiaojin Zhu,et al.  Kernel Regression with Order Preferences , 2007, AAAI.

[28]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[29]  Gianluca Pollastri,et al.  A neural network approach to ordinal regression , 2007, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[30]  Michael E. Tipping.  Sparse Bayesian Learning and the Relevance Vector Machine , 2001, J. Mach. Learn. Res..

[31]  Liangxiao Jiang,et al.  Augmenting naive Bayes for ranking , 2005, ICML.

[32]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[33]  Quoc V. Le,et al.  Learning to Rank with Nonsmooth Cost Functions , 2006, Neural Information Processing Systems.