Overcoming the Linearity of Ordinal Logistic Regression Adding Non-linear Covariates from Evolutionary Hybrid Neural Network Models

This paper proposes a non-linear ordinal logistic regression method based on the combination of a linear regression model and an evolutionary neural network with hybrid basis functions, combining Sigmoidal Unit and Radial Basis Functions neural networks. The process for obtaining the coefficients is carried out in several steps. Firstly we use an evolutionary algorithm to determine the structure of the hybrid neural network model, in a second step we augment the initial feature space covariate space adding the non-linear transformations of the input variables given by the hybrid hidden layer of the best individual of the evolutionary algorithm. Finally, we apply an ordinal logistic regression in the new feature space. This methodology is tested using 10 benchmark problems from the UCI repository. The hybrid model outperforms both the RBF and the SU pure models obtaining a good compromise between them and better results in terms of accuracy and ordinal classification error.

[1]  Michael Schmitt,et al.  On the Complexity of Computing and Learning with Multiplicative Neural Networks , 2002, Neural Computation.

[2]  Pedro Antonio Gutiérrez,et al.  Logistic Regression by Means of Evolutionary Radial Basis Function Neural Networks , 2011, IEEE Transactions on Neural Networks.

[3]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[4]  Bernhard Sick,et al.  Evolutionary optimization of radial basis function classifiers for data mining applications , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[5]  Christian Igel,et al.  Empirical evaluation of the improved Rprop learning algorithms , 2003, Neurocomputing.

[6]  Andrea Esuli,et al.  Evaluation Measures for Ordinal Regression , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[7]  Bernhard Schölkopf,et al.  A tutorial on support vector regression , 2004, Stat. Comput..

[8]  I. Johnstone,et al.  Projection-Based Approximation and a Duality with Kernel Methods , 1989 .

[9]  M. Friedman.  A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings , 1940 .

[10]  Shie-Jue Lee,et al.  An ART-based construction of RBF networks , 2002, IEEE Trans. Neural Networks.

[11]  Minqiang Li,et al.  Learning Subspace-Based RBFNN Using Coevolutionary Algorithm for Complex Classification Tasks , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[12]  Wei Chu,et al.  Support Vector Ordinal Regression , 2007, Neural Computation.

[13]  R. Tibshirani,et al.  Generalized additive models for medical research , 1986, Statistical methods in medical research.

[14]  César Hervás-Martínez,et al.  Evolutionary product unit based neural networks for regression , 2006, Neural Networks.

[15]  Pedro Antonio Gutiérrez,et al.  Combined projection and kernel basis functions for classification in evolutionary neural networks , 2009, Neurocomputing.

[16]  I. Soltesz.  Diversity in the Neuronal Machine: Order and Variability in Interneuronal Microcircuits , 2005 .

[17]  Tomás Maul,et al.  Early experiments with neural diversity machines , 2013, Neurocomputing.

[18]  P. McCullagh.  Regression Models for Ordinal Data , 1980 .

[19]  Nathan Intrator,et al.  A Hybrid Projection Based and Radial Basis Function Architecture , 2000, Multiple Classifier Systems.

[20]  Pedro Antonio Gutiérrez,et al.  Ordinal Classification Using Hybrid Artificial Neural Networks with Projection and Kernel Basis Functions , 2012, HAIS.

[21]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[22]  Chris Bishop,et al.  Improving the Generalization Properties of Radial Basis Function Neural Networks , 1991, Neural Computation.

[23]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[24]  R.P. Lippmann,et al.  Pattern classification using neural networks , 1989, IEEE Communications Magazine.

[25]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.