Algorithm Selection for Deep Active Learning with Imbalanced Datasets

Label efficiency has become an increasingly important objective in deep learning applications. Active learning aims to reduce the number of labeled examples needed to train deep networks, but the empirical performance of active learning algorithms can vary dramatically across datasets and applications. It is difficult to know in advance which active learning strategy will perform well or best in a given application. To address this, we propose the first adaptive algorithm selection strategy for deep active learning. For any unlabeled dataset, our (meta) algorithm TAILOR (Thompson ActIve Learning algORithm selection) iteratively and adaptively chooses among a set of candidate active learning algorithms. TAILOR uses novel reward functions aimed at gathering class-balanced examples. Extensive experiments in multi-class and multi-label applications demonstrate TAILOR's effectiveness in achieving accuracy comparable or better than that of the best of the candidate algorithms.

[1]  Wireless Communications and Mobile Computing.  Retracted: Active Learning for Imbalanced Data: The Difficulty and Proportions of Class Matter , 2023, Wireless Communications and Mobile Computing.

[2]  Manning Wang,et al.  Deep active learning models for imbalanced image classification , 2022, Knowl. Based Syst..

[3]  Danica J. Sutherland,et al.  Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernels , 2022, NeurIPS.

[4]  Kun Qian,et al.  Multi-label active learning through serial-parallel neural networks , 2022, Knowl. Based Syst..

[5]  Antoni B. Chan,et al.  A Comparative Survey of Deep Active Learning , 2022, ArXiv.

[6]  Alejandro Ribeiro,et al.  A Lagrangian Duality Approach to Active Learning , 2022, NeurIPS.

[7]  R. Nowak,et al.  GALAXY: Graph-based Active Learning at the Extreme , 2022, ICML.

[8]  Wojciech Czaja,et al.  Active Learning at the ImageNet Scale , 2021, ArXiv.

[9]  Hanghang Tong,et al.  Deep Active Learning by Leveraging Training Dynamics , 2021, NeurIPS.

[10]  Wolfgang Lehner,et al.  ImitAL: Learning Active Learning Strategies from Synthetic Data , 2021, ArXiv.

[11]  Claudio Gentile,et al.  Batch Active Learning at Scale , 2021, NeurIPS.

[12]  Suraj Kothawade,et al.  SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarios , 2021, NeurIPS.

[13]  Akshay Krishnamurthy,et al.  Gone Fishing: Neural Active Learning with Fisher Embeddings , 2021, NeurIPS.

[14]  Rishabh K. Iyer,et al.  Effective Evaluation of Deep Active Learning on Image Classification Tasks , 2021, ArXiv.

[15]  Kevin G. Jamieson,et al.  Learning to Actively Learn: A Robust Approach , 2020, ArXiv.

[16]  Christopher Mutschler,et al.  IALE: Imitating Active Learner Ensembles , 2020, J. Mach. Learn. Res..

[17]  Csaba Szepesvari,et al.  Bandit Algorithms , 2020 .

[18]  Cody A. Coleman,et al.  Similarity Search for Efficient Active Learning and Search of Rare Concepts , 2020, AAAI.

[19]  Jian Wu,et al.  Multi-Label Active Learning Algorithms for Image Classification , 2020, ACM Comput. Surv..

[20]  Céline Hudelot,et al.  Active Learning for Imbalanced Datasets , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[21]  Qing Wang,et al.  Learning to Sample: An Active Learning Framework , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[22]  Shai Shalev-Shwartz,et al.  Discriminative Active Learning , 2019, ArXiv.

[23]  John Langford,et al.  Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds , 2019, ICLR.

[24]  Alex Lamb,et al.  Deep Learning for Classical Japanese Literature , 2018, ArXiv.

[25]  Sethuraman Panchanathan,et al.  Multi-Label Deep Active Learning with Label Correlation , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[26]  Yang Wu,et al.  Dynamic Ensemble Active Learning: A Non-Stationary Bandit with Expert Advice , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[27]  Andreas Nürnberger,et al.  The Power of Ensembles for Active Learning in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Frédéric Precioso,et al.  Adversarial Active Learning for Deep Networks: a Margin Based Approach , 2018, ArXiv.

[29]  Ran El-Yaniv,et al.  Deep Active Learning over the Long Tail , 2017, ArXiv.

[30]  Silvio Savarese,et al.  Active Learning for Convolutional Neural Networks: A Core-Set Approach , 2017, ICLR.

[31]  Pascal Fua,et al.  Learning Active Learning from Data , 2017, NIPS.

[32]  Zoubin Ghahramani,et al.  Deep Bayesian Active Learning with Image Data , 2017, ICML.

[33]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Hsuan-Tien Lin,et al.  Active Learning by Learning , 2015, AAAI.

[35]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[36]  Pengpeng Zhao,et al.  Multi-label active learning for image classification , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[37]  Christian Igel,et al.  Active learning with support vector machines , 2014, WIREs Data Mining Knowl. Discov..

[38]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[39]  Jonathan Krause,et al.  3D Object Representations for Fine-Grained Categorization , 2013, 2013 IEEE International Conference on Computer Vision Workshops.

[40]  Benjamin Van Roy,et al.  Learning to Optimize via Posterior Sampling , 2013, Math. Oper. Res..

[41]  Csaba Szepesvári,et al.  Improved Algorithms for Linear Stochastic Bandits , 2011, NIPS.

[42]  Christopher K. I. Williams,et al.  International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge , 2022 .

[43]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Paul N. Bennett,et al.  Dual Strategy Active Learning , 2007, ECML.

[45]  G. Griffin,et al.  Caltech-256 Object Category Dataset , 2007 .

[46]  Maria-Florina Balcan,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[47]  Jiebo Luo,et al.  Learning multi-label scene classification , 2004, Pattern Recognit..

[48]  Ran El-Yaniv,et al.  Online Choice of Active Learning Algorithms , 2003, J. Mach. Learn. Res..

[49]  Gunnar Rätsch,et al.  Active Learning with Support Vector Machines in the Drug Discovery Process , 2003, J. Chem. Inf. Comput. Sci..

[50]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[51]  Gunnar Rätsch,et al.  Active Learning in the Drug Discovery Process , 2001, NIPS.

[52]  W. R. Thompson.  ON THE LIKELIHOOD THAT ONE UNKNOWN PROBABILITY EXCEEDS ANOTHER IN VIEW OF THE EVIDENCE OF TWO SAMPLES , 1933 .

[53]  Roman Garnett,et al.  Efficient nonmyopic batch active search , 2018, NeurIPS.

[54]  Andrew Y. Ng,et al.  Reading Digits in Natural Images with Unsupervised Feature Learning , 2011 .

[55]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[56]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .