On the Complexity of Learning From Counterexamples (extended abstract )

The complexity of learning concepts C E C from various concrete concept classes C 2x over a finite domain X is analyzed in terms of the number of counterexamples that are needed in the worst case (we consider the deterministic learning model of Angluin, where the learning algorithm produces a series of “equivalence queries”). It turns out that for many interesting concept classes C there exist exponential differences between the number of counterexamples that are required by a “naive” learning algorithm for C (e.g. one that always outputs the minimal consistent hypothesis) and a “smart” learning algorithm for C that attempts to make a more sophisticated prediction (this is in contrast to the situation for pac-learning, where every consistent learning algorithm requires about the same number of examples). We give O(logn) bounds for the number of counterexamples that are required for learning boxes, balls, and halfspaces in a d-dimensional discrete space X = { 1,. . . , n}d (for every finite dimension d). We also give an upper bound of O(@) and a lower bound of Q(d2) for the complexity of learning a threshold function with d input bits (i.e. X = {O, l}d). For each of these concept classes one can give learning algorithms that are both optimal (resp. close to optimal in the case of threshold functions) with regard to the number of counterexamples which they require computationally feasible (in the case of balls, halfspaces and threshold functions our learning algorithms use the method of the ellipsoid algorithm). Finally, we determine the complexity of learning of the considered concept classes (as well as linear orders, perfect matchings, and some other concept classes that turn out to be useful for the separation of learning models) on several variations of the considered learning model (such as learning with arbitrary hypotheses, partial hypotheses, membership queries). We also clarify the relationship between these learning models and some related combinatorial invariants.