Understanding the Effects of Batching in Online Active Learning

Online active learning (AL) algorithms often assume immediate access to a label once a query has been made. However, due to practical constraints, the labels of these queried examples are generally only available in “batches”. In this work, we present an analysis for a generic class of batch online AL algorithms, which reveals that the e↵ects of batching are in fact mild and only result in an additional label complexity term that is quasilinear in the batch size. To our knowledge, this provides the first theoretical justification for such algorithms and we show how they can be applied to batch variants of three canonical online AL algorithms: iwal, oriwal, and dhm. Finally, we also present empirical results across several benchmark datasets that corroborate these theoretical insights.

[1]  John Langford,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[2]  Dale Schuurmans,et al.  Discriminative Batch Mode Active Learning , 2007, NIPS.

[3]  Masashi Sugiyama,et al.  Improving importance estimation in pool-based batch active learning for approximate linear regression , 2012, Neural Networks.

[4]  S. Janson.  Tail bounds for sums of geometric and exponential variables , 2017, 1709.08157.

[5]  Andreas Krause,et al.  Near-optimal Batch Mode Active Learning and Adaptive Submodular Optimization , 2013, ICML.

[6]  Maria-Florina Balcan,et al.  The Power of Localization for Efficiently Learning Linear Separators with Noise , 2013, J. ACM.

[7]  Maria-Florina Balcan,et al.  Efficient Learning of Linear Separators under Bounded Noise , 2015, COLT.

[8]  Steve Hanneke,et al.  A bound on the label complexity of agnostic active learning , 2007, ICML '07.

[9]  Rong Jin,et al.  Batch mode active learning and its application to medical image classification , 2006, ICML.

[10]  Rong Jin,et al.  Large-scale text categorization by batch mode active learning , 2006, WWW '06.

[11]  John Langford,et al.  Agnostic Active Learning Without Constraints , 2010, NIPS.

[12]  Yi Zhang,et al.  Incorporating Diversity and Density in Active Learning for Relevance Feedback , 2007, ECIR.

[13]  Adam Tauman Kalai,et al.  Analysis of Perceptron-Based Active Learning , 2009, COLT.

[14]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[15]  Sanjoy Dasgupta,et al.  A General Agnostic Active Learning Algorithm , 2007, ISAIM.

[16]  John Langford,et al.  Importance weighted active learning , 2008, ICML '09.

[17]  Maria-Florina Balcan,et al.  Margin Based Active Learning , 2007, COLT.

[18]  Kamal Nigamyknigam,et al.  Employing Em in Pool-based Active Learning for Text Classiication , 1998 .

[19]  Klaus Brinker,et al.  Incorporating Diversity in Active Learning with Support Vector Machines , 2003, ICML.

[20]  Sanjoy Dasgupta,et al.  Hierarchical sampling for active learning , 2008, ICML '08.

[21]  Rong Jin,et al.  Semi-supervised SVM batch mode active learning for image retrieval , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  András György,et al.  Online Learning under Delayed Feedback , 2013, ICML.

[23]  Chicheng Zhang,et al.  Efficient active learning of sparse halfspaces , 2018, COLT.

[24]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[25]  Claudio Gentile,et al.  Region-Based Active Learning , 2019, AISTATS.

[26]  Claudio Gentile,et al.  Active Learning with Disagreement Graphs , 2019, ICML.