Multiclass Bounded Logistic Regression { Ecient Regularization with Interior Point Method

Logistic regression has been widely used in classification tasks for many years. Its optimization in case of linear separable data has received extensive study due to the problem of a monoton likelihood. This paper presents a new approach, called bounded logistic regression (BLR), by solving the logistic regression as a convex optimization problem with constraints. The paper tests the accuracy of BLR by evaluating nine well-known datasets and compares it to the closely related support vector machine approach (SVM).

[1]  Chih-Jen Lin,et al.  Trust Region Newton Method for Logistic Regression , 2008, J. Mach. Learn. Res..

[2]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[3]  Koby Crammer,et al.  On the Learnability and Design of Output Codes for Multiclass Problems , 2002, Machine Learning.

[4]  Michael E. Tipping.  The Relevance Vector Machine , 1999, NIPS.

[5]  J. Pang,et al.  Computational optimization : a tribute to Olvi Mangasarian , 1999 .

[6]  Wei-Yin Loh,et al.  A Comparison of Prediction Accuracy, Complexity, and Training Time of Thirty-Three Old and New Classification Algorithms , 2000, Machine Learning.

[7]  Stephen P. Boyd,et al.  An Interior-Point Method for Large-Scale $\ell_1$-Regularized Least Squares , 2007, IEEE Journal of Selected Topics in Signal Processing.

[8]  K. Bennett,et al.  A support vector machine approach to decision trees , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[9]  U. Alon,et al.  Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Jason Weston,et al.  Multi-Class Support Vector Machines , 1998 .

[11]  Thomas J. Santner,et al.  A note on A. Albert and J. A. Anderson's conditions for the existence of maximum likelihood estimates in logistic regression models , 1986 .

[12]  Jan Paul Siebert,et al.  Vehicle Recognition Using Rule Based Methods , 1987 .

[13]  Simon Haykin,et al.  Selected topics in signal processing , 1989 .

[14]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[15]  Tao Yu,et al.  High-dimensional pseudo-logistic regression and classification with applications to gene expression data , 2007, Comput. Stat. Data Anal..

[16]  Andreas Antoniou,et al.  Practical Optimization: Algorithms and Engineering Applications , 2007, Texts in Computer Science.

[17]  Chih-Jen Lin,et al.  A comparison of methods for multiclass support vector machines , 2002, IEEE Trans. Neural Networks.

[18]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[19]  Ole Winther,et al.  Gaussian processes and SVM: Mean field and leave-one-out estimator , 2000 .

[20]  A. Albert,et al.  On the existence of maximum likelihood estimates in logistic regression models , 1984 .