Deterministic Online Classification: Non-iteratively Reweighted Recursive Least-Squares for Binary Class Rebalancing

Deterministic solutions are becoming more critical for interpretability. Weighted Least-Squares (WLS) has been widely used as a deterministic batch solution with a specific weight design. In the online settings of WLS, exact reweighting is necessary to converge to its batch settings. In order to comply with its necessity, the iteratively reweighted least-squares algorithm is mainly uti-lized with a linearly growing time complexity which is not attractive for online learning. Due to the high and growing computational costs, an efficient online formulation of reweighted least-squares is desired. We introduce a new deterministic online classification algorithm of WLS with a constant time complexity for binary class rebalancing. We demonstrate that our proposed online formulation exactly converges to its batch formulation and outperforms existing state-of-the-art stochastic online binary classification algorithms in real-world data sets empirically.

[1]  Pradeep Ravikumar,et al.  Class-Weighted Classification: Trade-offs and Robust Approaches , 2020, ICML.

[2]  Andrew Beng Jin Teoh,et al.  Online Heterogeneous Face Recognition Based on Total-Error-Rate Minimization , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[3]  Chunyan Miao,et al.  Second-Order Online Active Learning and Its Applications , 2018, IEEE Transactions on Knowledge and Data Engineering.

[4]  Min Wu,et al.  Adaptive Cost-Sensitive Online Classification , 2018, IEEE Transactions on Knowledge and Data Engineering.

[5]  Steven C. H. Hoi,et al.  Online Learning: A Comprehensive Survey , 2018, Neurocomputing.

[6]  Ling Jian,et al.  Budget Online Learning Algorithm for Least Squares SVM , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[8]  Giorgio Metta,et al.  Incremental robot learning of new objects with fixed update time , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Changshui Zhang,et al.  Dependent Online Kernel Learning With Constant Number of Random Fourier Features , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[10]  Kar-Ann Toh,et al.  Exploiting the relationships among several binary classifiers via data transformation , 2014, Pattern Recognit..

[11]  Andrew Beng Jin Teoh,et al.  An online learning network for biometric scores fusion , 2013, Neurocomputing.

[12]  C. Scott.  Calibrated asymmetric surrogate losses , 2012 .

[13]  Andrew Beng Jin Teoh,et al.  An online AUC formulation for binary classification , 2012, Pattern Recognit..

[14]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[15]  Hao Helen Zhang,et al.  Robust Model-Free Multiclass Probability Estimation , 2010, Journal of the American Statistical Association.

[16]  Koby Crammer,et al.  Adaptive regularization of weight vectors , 2009, Machine Learning.

[17]  Kar-Ann Toh,et al.  Deterministic Neural Classification , 2008, Neural Computation.

[18]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[19]  Kar-Ann Toh,et al.  Between Classification-Error Approximation and Weighted Least-Squares Learning , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[21]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[22]  Koby Crammer,et al.  Online Passive-Aggressive Algorithms , 2003, J. Mach. Learn. Res..

[23]  Yann LeCun,et al.  Large Scale Online Learning , 2003, NIPS.

[24]  Charles Elkan,et al.  The Foundations of Cost-Sensitive Learning , 2001, IJCAI.

[25]  Steve Rogers,et al.  Adaptive Filter Theory , 1996 .

[26]  S. Stigler.  Gauss and the Invention of Least Squares , 1981 .

[27]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[28]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[29]  Seetha Hari,et al.  Learning From Imbalanced Data , 2019, Advances in Computer and Electrical Engineering.

[30]  Brandon M. Greenwell,et al.  Interpretable Machine Learning , 2019, Hands-On Machine Learning with R.

[31]  Kim-Chuan Toh,et al.  A Unified Formulation and Fast Accelerated Proximal Gradient Method for Classification , 2017, J. Mach. Learn. Res..

[32]  Steven C. H. Hoi,et al.  Large Scale Online Kernel Learning , 2016, J. Mach. Learn. Res..

[33]  Shai Shalev-Shwartz,et al.  Online learning: theory, algorithms and applications (למידה מקוונת.) , 2007 .

[34]  J. Willems.  Deterministic least squares filtering , 2004 .

[35]  S. R. Searle,et al.  On Deriving the Inverse of a Sum of Matrices , 1981 .