Online Agnostic Boosting via Regret Minimization

Boosting is a widely used machine learning approach based on the idea of aggregating weak learning rules. While in statistical learning numerous boosting methods exist both in the realizable and agnostic settings, in online learning they exist only in the realizable case. In this work we provide the first agnostic online boosting algorithm; that is, given a weak learner with only marginally-better-than-trivial regret guarantees, our algorithm boosts it to a strong learner with sublinear regret. Our algorithm is based on an abstract (and simple) reduction to online convex optimization, which efficiently converts an arbitrary online convex optimizer to an online booster. Moreover, this reduction extends to the statistical as well as the online realizable settings, thus unifying the 4 cases of statistical/online and agnostic/realizable boosting.

[1]  Naman Agarwal,et al.  Boosting for Dynamical Systems , 2019, ArXiv.

[2]  Hsuan-Tien Lin,et al.  An Online Boosting Algorithm with Theoretical Justifications , 2012, ICML.

[3]  Noga Alon,et al.  Private PAC learning implies finite Littlestone dimension , 2018, STOC.

[4]  Roi Livni,et al.  An Equivalence Between Private Classification and Online Prediction , 2020, 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS).

[5]  Shai Ben-David,et al.  Agnostic Boosting , 2001, COLT/EuroCOLT.

[6]  Karan Singh,et al.  The Price of Differential Privacy for Online Learning , 2017, ICML.

[7]  Haipeng Luo,et al.  Optimal and Adaptive Algorithms for Online Boosting , 2015, ICML.

[8]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[9]  Dmitry Gavinsky.  Optimally-Smooth Adaptive Boosting and Application to Agnostic Learning , 2003, J. Mach. Learn. Res..

[10]  Maria-Florina Balcan,et al.  Communication Efficient Distributed Agnostic Boosting , 2015, AISTATS.

[11]  Y. Freund,et al.  Adaptive game playing using multiplicative weights , 1999 .

[12]  Roi Livni,et al.  Passing Tests without Memorizing: Two Models for Fooling Discriminators , 2019, ArXiv.

[13]  Shay Moran,et al.  Supervised learning through the lens of compression , 2016, NIPS.

[14]  Rocco A. Servedio,et al.  Boosting in the presence of noise , 2003, STOC '03.

[15]  Karan Singh,et al.  Logarithmic Regret for Online Control , 2019, NeurIPS.

[16]  Sham M. Kakade,et al.  The Nonstochastic Control Problem , 2020, ALT.

[17]  Shay Moran,et al.  Private Learning Implies Online Learning: An Efficient Reduction , 2019, NeurIPS.

[18]  Haipeng Luo,et al.  Online Gradient Boosting , 2015, NIPS.

[19]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[20]  M. Sion.  On general minimax theorems , 1958 .

[21]  Sham M. Kakade,et al.  Online Control with Adversarial Disturbances , 2019, ICML.

[22]  Vitaly Feldman,et al.  Distribution-Specific Agnostic Boosting , 2009, ICS.

[23]  Yoav Freund,et al.  Game theory, on-line prediction and boosting , 1996, COLT '96.

[24]  Rocco A. Servedio,et al.  Adaptive Martingale Boosting , 2008, NIPS.

[25]  Adam Tauman Kalai,et al.  Potential-Based Agnostic Boosting , 2009, NIPS.

[26]  Yoav Freund,et al.  Boosting a weak learning algorithm by majority , 1995, COLT '90.

[27]  Adam Tauman Kalai,et al.  On agnostic boosting and parity learning , 2008, STOC.

[28]  R. Schapire.  The Strength of Weak Learnability , 1990, Machine Learning.

[29]  Alex M. Andrew,et al.  Boosting: Foundations and Algorithms , 2012 .

[30]  Elad Hazan,et al.  Introduction to Online Convex Optimization , 2016, Found. Trends Optim..

[31]  Yishay Mansour,et al.  Boosting Using Branching Programs , 2000, J. Comput. Syst. Sci..

[32]  Seth Neel,et al.  The Role of Interactivity in Local Differential Privacy , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[33]  Seth Neel,et al.  How to Use Heuristics for Differential Privacy , 2019, 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS).

[34]  Ambuj Tewari,et al.  Online Learning via Differential Privacy , 2017, ArXiv.