Support Vector Machine Active Learning with Applications to Text Classification

Support vector machines have met with significant success in numerous real-world learning tasks. However, like most machine learning algorithms, they are generally applied using a randomly selected training set classified in advance. In many settings, we also have the option of using pool-based active learning. Instead of using a randomly selected training set, the learner has access to a pool of unlabeled instances and can request the labels for some number of them. We introduce a new algorithm for performing active learning with support vector machines, i.e., an algorithm for choosing which instances to request next. We provide a theoretical motivation for the algorithm using the notion of a version space. We present experimental results showing that employing our active learning method can significantly reduce the need for labeled training instances in both the standard inductive and transductive settings.

[1]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[2]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[3]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[4]  Tom M. Mitchell,et al.  Generalization as Search , 2002 .

[5]  Jean-Claude Latombe,et al.  Robot motion planning , 1970, The Kluwer international series in engineering and computer science.

[6]  Eric Horvitz,et al.  Time-Dependent Utility and Action Under Uncertainty , 1991, UAI.

[7]  H. Sebastian Seung,et al.  Query by committee , 1992, COLT '92.

[8]  David Heckerman,et al.  Troubleshooting Under Uncertainty , 1994 .

[9]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[10]  David D. Lewis,et al.  Heterogeneous Uncertainty Sampling for Supervised Learning , 1994, ICML.

[11]  Ken Lang,et al.  NewsWeeder: Learning to Filter Netnews , 1995, ICML.

[12]  Shlomo Argamon,et al.  Committee-Based Sampling For Training Probabilistic Classi(cid:12)ers , 1995 .

[13]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[14]  Prasad Tadepalli,et al.  Active Learning with Committees for Text Categorization , 1997, AAAI/IAAI.

[15]  Susan T. Dumais,et al.  Inductive learning algorithms and representations for text categorization , 1998, CIKM '98.

[16]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[17]  Andrew McCallum,et al.  Employing EM and Pool-Based Active Learning for Text Classification , 1998, ICML.

[18]  Thorsten Joachims,et al.  Text Categorization with Support Vector Machines: Learning with Many Relevant Features , 1998, ECML.

[19]  J. C. BurgesChristopher.  A Tutorial on Support Vector Machines for Pattern Recognition , 1998 .

[20]  Kamal Nigamyknigam,et al.  Employing Em in Pool-based Active Learning for Text Classiication , 1998 .

[21]  Thorsten Joachims,et al.  Making large scale SVM learning practical , 1998 .

[22]  Thorsten Joachims,et al.  Text categorization with support vector machines , 1999 .

[23]  David A. McAllester.  PAC-Bayesian model averaging , 1999, COLT '99.

[24]  Ralf Herbrich,et al.  Bayes Point Machines: Estimating the Bayes Point in Kernel Space , 1999 .

[25]  Nello Cristianini,et al.  Further results on the margin distribution , 1999, COLT '99.

[26]  Thorsten Joachims,et al.  Transductive Inference for Text Classification using Support Vector Machines , 1999, ICML.

[27]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorisation: a survey , 1999 .

[28]  B. Schölkopf,et al.  Advances in kernel methods: support vector learning , 1999 .

[29]  Nello Cristianini,et al.  Query Learning with Large Margin Classi ersColin , 2000 .

[30]  Greg Schohn,et al.  Less is More: Active Learning with Support Vector Machines , 2000, ICML.

[31]  Prasad Tadepalli,et al.  Active learning with committees: an approach to efficient learning in text categorization using linear threshold algorithms , 2000 .

[32]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[33]  Colin Campbell,et al.  Bayes Point Machines , 2001, J. Mach. Learn. Res..

[34]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.