A new approach in model selection for ordinal target variables

This paper introduces a novel approach to assess model performance for predictive models characterized by an ordinal target variable in order to satisfy the lack of suitable tools in this framework. Our methodological proposal is a new index for model assessment which satisfies mathematical properties and can be easily computed. In order to show how our performance indicator works, empirical evidence achieved on a toy examples and simulated data are provided. On the basis of results at hand, we underline that our approach discriminates better for model selection with respect to performance indexes proposed in the literature.

[1]  Josep Domingo-Ferrer,et al.  Regression for ordinal variables without underlying continuous variables , 2006, Inf. Sci..

[2]  A. Piscitelli,et al.  How Disadvantages Shape Life Satisfaction: An Alternative Methodological Approach , 2019 .

[3]  David J. Hand,et al.  Measuring Diagnostic Accuracy of Statistical Prediction Rules , 2001 .

[4]  Andrew P. Bradley,et al.  The use of the area under the ROC curve in the evaluation of machine learning algorithms , 1997, Pattern Recognit..

[5]  Bo Pang,et al.  Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales , 2005, ACL.

[6]  A. Agresti.  Analysis of Ordinal Categorical Data: Agresti/Analysis , 2010 .

[7]  Giuliano Galimberti,et al.  Classification Trees for Ordinal Responses in R: The rpartScore Package , 2012 .

[8]  Nathalie Japkowicz,et al.  Evaluation Methods for Ordinal Classification , 2009, Canadian Conference on AI.

[9]  Ordinal Classification Trees Based on Impurity Measures , 2004 .

[10]  Silvia Figini,et al.  Computational Statistics and Data Analysis Making Classifier Performance Comparisons When Roc Curves Intersect , 2022 .

[11]  David J. Hand,et al.  Measuring classifier performance: a coherent alternative to the area under the ROC curve , 2009, Machine Learning.

[12]  Roman Hornung,et al.  Ordinal Forests , 2020, J. Classif..

[13]  David J. Hand,et al.  A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems , 2001, Machine Learning.

[14]  David J. Hand,et al.  Construction and Assessment of Classification Rules , 1997 .

[15]  Bernard De Baets,et al.  ROC analysis in ordinal regression learning , 2008, Pattern Recognit. Lett..

[16]  J. Hanley,et al.  The meaning and use of the area under a receiver operating characteristic (ROC) curve. , 1982, Radiology.

[17]  Niall M. Adams,et al.  Improving the Practice of Classifier Performance Assessment , 2000, Neural Computation.

[18]  M. N. Sulaiman,et al.  A Review On Evaluation Metrics For Data Classification Evaluations , 2015 .

[19]  Wojciech Kotlowski,et al.  Stochastic dominance-based rough set model for ordinal classification , 2008, Inf. Sci..

[20]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[21]  Wei Chen,et al.  Toward a Better Understanding of Model Validation Metrics , 2011 .

[22]  Jaime S. Cardoso,et al.  Measuring the Performance of Ordinal Classification , 2011, Int. J. Pattern Recognit. Artif. Intell..

[23]  Charles X. Ling,et al.  Using AUC and accuracy in evaluating learning algorithms , 2005, IEEE Transactions on Knowledge and Data Engineering.

[24]  Charles X. Ling,et al.  Constructing New and Better Evaluation Measures for Machine Learning , 2007, IJCAI.

[25]  Gavin Brown,et al.  Random Ordinality Ensembles: Ensemble methods for multi-valued categorical data , 2015, Inf. Sci..

[26]  Eibe Frank,et al.  A Simple Approach to Ordinal Classification , 2001, ECML.

[27]  Steven L. Salzberg.  On Comparing Classifiers: A Critique of Current Research and Methods , 1999 .

[28]  Stan Szpakowicz,et al.  Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation , 2006, Australian Conference on Artificial Intelligence.