Active Learning from Weak and Strong Labelers

An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses active learning with labels obtained from strong and weak labelers, where in addition to the standard active learning setting, we have an extra weak labeler which may occasionally provide incorrect labels. An example is learning to classify medical images where either expensive labels may be obtained from a physician (oracle or strong labeler), or cheaper but occasionally incorrect labels may be obtained from a medical resident (weak labeler). Our goal is to learn a classifier with low error on data labeled by the oracle, while using the weak labeler to reduce the number of label queries made to this labeler. We provide an active learning algorithm for this setting, establish its statistical consistency, and analyze its label complexity to characterize when it can provide label savings over using the strong labeler alone.

[1]  John Langford,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[2]  Shai Ben-David,et al.  New England , 1894, Letters from America.

[3]  Robert D. Nowak,et al.  The Geometry of Generalized Binary Search , 2009, IEEE Transactions on Information Theory.

[4]  Xindong Wu,et al.  Self-Taught Active Learning from Crowds , 2012, 2012 IEEE 12th International Conference on Data Mining.

[5]  Adam Tauman Kalai,et al.  Reliable Agnostic Learning , 2009, COLT.

[6]  Daniel J. Hsu.  Algorithms for active learning , 2010 .

[7]  Nader H. Bshouty,et al.  Maximizing Agreements with One-Sided Error with Applications to Heuristic Learning , 2005, Machine Learning.

[8]  Kristen Grauman,et al.  What's it going to cost you?: Predicting effort vs. informativeness for multi-label image annotations , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[9]  Anand D. Sarwate,et al.  Learning from Data with Heterogeneous Noise using SGD , 2014, AISTATS.

[10]  John Langford,et al.  Active Learning with an ERM Oracle , 2009 .

[11]  Hans Ulrich Simon.  PAC-learning in the presence of one-sided classification noise , 2012, Annals of Mathematics and Artificial Intelligence.

[12]  John Langford,et al.  Agnostic Active Learning Without Constraints , 2010, NIPS.

[13]  Sanjoy Dasgupta,et al.  A General Agnostic Active Learning Algorithm , 2007, ISAIM.

[14]  Claudio Gentile,et al.  Selective sampling and active learning from single and multiple teachers , 2012, J. Mach. Learn. Res..

[15]  Maria-Florina Balcan,et al.  Active and passive learning of linear separators under log-concave distributions , 2012, COLT.

[16]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[17]  Panagiotis G. Ipeirotis,et al.  Repeated labeling using multiple noisy labelers , 2012, Data Mining and Knowledge Discovery.

[18]  Sanjoy Dasgupta,et al.  Coarse sample complexity bounds for active learning , 2005, NIPS.

[19]  Jennifer G. Dy,et al.  Active Learning from Crowds , 2011, ICML.

[20]  Jennifer G. Dy,et al.  Active Learning from Multiple Knowledge Sources , 2012, AISTATS.

[21]  Varun Kanade,et al.  Distribution-independent Reliable Learning , 2014, COLT.

[22]  Jaime G. Carbonell,et al.  Proactive learning: cost-sensitive active learning with multiple imperfect oracles , 2008, CIKM '08.

[23]  Mausam,et al.  To Re(label), or Not To Re(label) , 2014, HCOMP.

[24]  Kristen Grauman,et al.  Cost-Sensitive Active Visual Category Learning , 2010, International Journal of Computer Vision.

[25]  Koby Crammer,et al.  Learning Bounds for Domain Adaptation , 2007, NIPS.

[26]  C. H. Lin.  Reactive Learning : Actively Trading Off Larger Noisier Training Sets Against Smaller Cleaner Ones , 2015 .

[27]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[28]  Steve Hanneke,et al.  A bound on the label complexity of agnostic active learning , 2007, ICML '07.

[29]  Koby Crammer,et al.  Learning from Data of Variable Quality , 2005, NIPS.

[30]  Kamalika Chaudhuri,et al.  Beyond Disagreement-Based Agnostic Active Learning , 2014, NIPS.