Selective Sampling Using the Query by Committee Algorithm

We analyze the “query by committee” algorithm, a method for filtering informative queries from a random stream of inputs. We show that if the two-member committee algorithm achieves information gain with positive lower bound, then the prediction error decreases exponentially with the number of queries. We show that, in particular, this exponential decrease holds for query learning of perceptrons.

[1]  D. Lindley.  On a Measure of the Information Provided by an Experiment , 1956 .

[2]  W. J. Studden,et al.  Theory Of Optimal Experiments , 1972 .

[3]  Norbert Sauer,et al.  On the Density of Families of Sets , 1972, J. Comb. Theory, Ser. A.

[4]  Tom Michael Mitchell.  Version spaces: an approach to concept learning. , 1979 .

[5]  Tom M. Mitchell,et al.  Generalization as Search , 2002 .

[6]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[7]  Peter Smith.  Convexity methods in variational calculus , 1985 .

[8]  Leo F. Boron,et al.  Theory of Convex Bodies , 1988 .

[9]  Dana Angluin,et al.  Queries and concept learning , 1988, Machine Learning.

[10]  Colin McDiarmid,et al.  Surveys in Combinatorics, 1989: On the method of bounded differences , 1989 .

[11]  David A. Cohn,et al.  Training Connectionist Networks with Queries and Selective Sampling , 1989, NIPS.

[12]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[13]  Ronald L. Rivest,et al.  On the sample complexity of pac-learning using random and chosen examples , 1990, Annual Conference Computational Learning Theory.

[14]  Wolfgang Kinzel,et al.  Improving a Network Generalization Ability by Selecting Examples , 1990 .

[15]  David Haussler,et al.  Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension , 1991, COLT '91.

[16]  Yuh-Dauh Lyuu,et al.  The Transition to Perfect Generalization in Perceptrons , 1991, Neural Computation.

[17]  Eric B. Baum,et al.  Neural net algorithms that learn in polynomial time from examples and queries , 1991, IEEE Trans. Neural Networks.

[18]  Michael Kearns,et al.  Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[19]  H. Sebastian Seung,et al.  Query by committee , 1992, COLT '92.

[20]  W. Näther.  Optimum experimental designs , 1994 .

[21]  P. Laycock,et al.  Optimum Experimental Designs , 1995 .

[22]  Shlomo Argamon,et al.  Committee-Based Sampling For Training Probabilistic Classi(cid:12)ers , 1995 .