Ordinal regression neural networks based on concentric hyperspheres

Threshold models are one of the most common approaches for ordinal regression, based on projecting patterns to the real line and dividing this real line in consecutive intervals, one interval for each class. However, finding such one-dimensional projection can be too harsh an imposition for some datasets. This paper proposes a multidimensional latent space representation with the purpose of relaxing this projection, where the different classes are arranged based on concentric hyperspheres, each class containing the previous classes in the ordinal scale. The proposal is implemented through a neural network model, each dimension being a linear combination of a common set of basis functions. The model is compared to a nominal neural network, a neural network based on the proportional odds model and to other state-of-the-art ordinal regression methods for a total of 12 datasets. The proposed latent space shows an improvement on the two performance metrics considered, and the model based on the three-dimensional latent space obtains competitive performance when compared to the other methods.

[1]  Qinghua Zheng,et al.  Ordinal extreme learning machine , 2010, Neurocomputing.

[2]  Wei Chu,et al.  Support Vector Ordinal Regression , 2007, Neural Computation.

[3]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[4]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[5]  M. Friedman A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[6]  Vladimir Pavlovic,et al.  Structured Output Ordinal Regression for Dynamic Facial Emotion Intensity Prediction , 2010, ECCV.

[7]  Ling Li,et al.  Reduction from Cost-Sensitive Ordinal Ranking to Weighted Binary Classification , 2012, Neural Computation.

[8]  P. McCullagh Regression Models for Ordinal Data , 1980 .

[9]  Xiaoming Zhang,et al.  Kernel Discriminant Learning for Ordinal Regression , 2010, IEEE Transactions on Knowledge and Data Engineering.

[10]  Xizhao Wang,et al.  Dynamic ensemble extreme learning machine based on sample entropy , 2012, Soft Comput..

[11]  María Pérez-Ortiz,et al.  An Experimental Study of Different Ordinal Regression Methods and Measures , 2012, HAIS.

[12]  Jaime S. Cardoso,et al.  The unimodal model for the classification of ordinal data , 2008, Neural Networks.

[13]  Yoram Singer,et al.  Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers , 2000, J. Mach. Learn. Res..

[14]  A. Agresti,et al.  Categorical Data Analysis , 1991, International Encyclopedia of Statistical Science.

[15]  Mario Costa,et al.  Probabilistic Interpretation of Feedforward Network Outputs, with Relationships to Statistical Prediction of Ordinal Quantities , 1996, Int. J. Neural Syst..

[16]  Pedro Antonio Gutiérrez,et al.  Metrics to guide a multi-objective evolutionary algorithm for ordinal classification , 2014, Neurocomputing.

[17]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[18]  Thomas G. Dietterich,et al.  Solving Multiclass Learning Problems via Error-Correcting Output Codes , 1994, J. Artif. Intell. Res..

[19]  Andrés Bueno-Crespo,et al.  Neural architecture design based on extreme learning machine , 2013, Neural Networks.

[20]  Ling Li,et al.  Ordinal Regression by Extended Binary Classification , 2006, NIPS.

[21]  Jaime S. Cardoso,et al.  Modelling ordinal relations with SVMs: An application to objective aesthetic evaluation of breast cancer conservative treatment , 2005, Neural Networks.

[22]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[23]  Jaime S. Cardoso,et al.  Classification of Ordinal Data Using Neural Networks , 2005, ECML.

[24]  Bernard De Baets,et al.  Learning partial ordinal class memberships with kernel-based proportional odds models , 2012, Comput. Stat. Data Anal..

[25]  Andrea Esuli,et al.  Evaluation Measures for Ordinal Regression , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[26]  Wang Xi-zhao,et al.  Architecture selection for networks trained with extreme learning machine using localized generalization error model , 2013 .

[27]  Gianluca Pollastri,et al.  A neural network approach to ordinal regression , 2007, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[28]  Jaime S. Cardoso,et al.  Learning to Classify Ordinal Data: The Data Replication Method , 2007, J. Mach. Learn. Res..

[29]  Annalisa Riccardi,et al.  Cost-Sensitive AdaBoost Algorithm for Ordinal Regression Based on Extreme Learning Machine , 2014, IEEE Transactions on Cybernetics.

[30]  Xizhao Wang,et al.  Architecture selection for networks trained with extreme learning machine using localized generalization error model , 2013, Neurocomputing.

[31]  Pedro Antonio Gutiérrez,et al.  Exploitation of Pairwise Class Distances for Ordinal Classification , 2013, Neural Computation.

[32]  G. G. Stokes "J." , 1890, The New Yale Book of Quotations.

[33]  Annalisa Riccardi,et al.  Ordinal Neural Networks Without Iterative Tuning , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[34]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[35]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[36]  Hui Wang,et al.  Ordinal regression with continuous pairwise preferences , 2012, Int. J. Mach. Learn. Cybern..

[37]  Mark Mathieson Ordered Classes and Incomplete Examples in Classification , 1996, NIPS.