Pool-based active learning in approximate linear regression

The goal of pool-based active learning is to choose the best input points to gather output values from a ‘pool’ of input samples. We develop two pool-based active learning criteria for linear regression. The first criterion allows us to obtain a closed-form solution so it is computationally very efficient. However, this solution is not necessarily optimal in the single-trial generalization error analysis. The second criterion can give a better solution, but it does not have a closed-form solution and therefore some additional search strategy is needed. To cope with this problem, we propose a practical procedure which enables us to efficiently search for a better solution around the optimal solution of the first method. Simulations with toy and benchmark datasets show that the proposed active learning method compares favorably with other active learning methods as well as the baseline passive learning scheme. Furthermore, the usefulness of the proposed active learning method is also demonstrated in wafer alignment in semiconductor exposure apparatus.

[1]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[2]  Mk Warmuth,et al.  Active Learning with SVMs in the Drug Discovery Process , 2003 .

[3]  Steffen Bickel,et al.  Discriminative learning for differing training and test distributions , 2007, ICML '07.

[4]  Takafumi Kanamori,et al.  Pool-based active learning with optimal sampling distribution and its information geometrical interpretation , 2007, Neurocomputing.

[5]  Masashi Sugiyama,et al.  Active Learning in Approximately Linear Regression Based on Conditional Expectation of Generalization Error , 2006, J. Mach. Learn. Res..

[6]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .

[7]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[8]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[9]  Masashi Sugiyama,et al.  A batch ensemble approach to active learning with model selection , 2008, Neural Networks.

[10]  D. Coomans,et al.  Comparison of Multivariate Discrimination Techniques for Clinical Data— Application to the Thyroid Functional State , 1983, Methods of Information in Medicine.

[11]  Raymond J. Mooney,et al.  Diverse ensembles for active learning , 2004, ICML.

[12]  Hidetoshi Shimodaira,et al.  Active learning algorithm using the maximum weighted log-likelihood estimator , 2003 .

[13]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[14]  Ronald L. Wasserstein,et al.  Monte Carlo: Concepts, Algorithms, and Applications , 1997 .

[15]  Masashi Sugiyama,et al.  Input-dependent estimation of generalization error under covariate shift , 2005 .

[16]  Stéphane Lecoeuche,et al.  SAKM: Self-adaptive kernel machine A kernel-based algorithm for online clustering , 2008, Neural Networks.

[17]  Kamal Nigamyknigam,et al.  Employing Em in Pool-based Active Learning for Text Classiication , 1998 .

[18]  D. Wiens Robust weights and designs for biased regression models: Least squares and generalized M-estimation , 2000 .

[19]  Masashi Sugiyama,et al.  Active Learning with Model Selection — Simultaneous Optimization of Sample Points and Models for Trigonometric Polynomial Models , 2003 .

[20]  Neil D. Lawrence,et al.  A Conditional Expectation Approach to Model Selection and Active Learning under Covariate Shift , 2009 .

[21]  Bernhard Schölkopf,et al.  Correcting Sample Selection Bias by Unlabeled Data , 2006, NIPS.

[22]  M. Kawanabe,et al.  Direct importance estimation for covariate shift adaptation , 2008 .

[23]  Klaus-Robert Müller,et al.  Covariate Shift Adaptation by Importance Weighted Cross Validation , 2007, J. Mach. Learn. Res..

[24]  Gunnar Rätsch,et al.  Active Learning with Support Vector Machines in the Drug Discovery Process , 2003, J. Chem. Inf. Comput. Sci..

[25]  Lyle H. Ungar,et al.  Machine Learning manuscript No. (will be inserted by the editor) Active Learning for Logistic Regression: , 2007 .

[26]  Andrew McCallum,et al.  Employing EM and Pool-Based Active Learning for Text Classification , 1998, ICML.

[27]  Ramon E. Henkel Tests of Significance , 1976 .

[28]  Thomas Hofmann,et al.  Active learning for misspecified generalized linear models , 2007 .

[29]  Kenji Fukumizu,et al.  Statistical active learning in multilayer perceptrons , 2000, IEEE Trans. Neural Networks Learn. Syst..

[30]  Pierre Baldi,et al.  Bioinformatics - the machine learning approach (2. ed.) , 2000 .

[31]  Geoffrey E. Hinton,et al.  The delve manual , 1996 .