Learning binary relations, total orders, and read-once formulas

We study learning problems under various models. We consider an on-line model in which the learner answers a sequence of yes/no questions with immediate feedback after each question. The complexity of the learning task depends on the agent selecting the sequence of questions. We present an extended mistake-bound model in which the sequence of questions is selected by a helpful teacher, by the learner, by an adversary, or at random. We study the problem of learning a relation between two sets of objects. If the relation has no structure, the learner cannot make good predictions. We impose structure by restricting one set of objects to have relatively few "types". We describe efficient algorithms to learn a binary relation for the various selection methods. Complementing these results are lower bounds, often proving that the algorithms perform optimally. We consider the problem of learning a total order on a set of elements. We restrict the predicate of the relation to be a total order. Both upper and lower bounds are provided for the different selection methods. We uncover an interesting relationship between learning theory and randomized approximation schemes. When a teacher selects the sequence of questions, we ask: what is the minimum number of examples a teacher must reveal to uniquely identify the target concept? It is a paradox that for many concept classes, the number of mistakes made with a helpful teacher may be worse than the number of mistakes made when the learner selects the sequence. When the learner chooses the sequence, the number of mistakes can be significantly smaller than the number of queries needed. We present a technique for exactly identifying read-once formulas from random examples. The method is based on sampling the input-output behavior of the target formula on a probability distribution which is determined by the fixed point of the formula's amplification function. Efficient algorithms exactly identify families of read-once formulas over various bases. We apply these results to prove the existence of polynomial-length universal identification sequences for large classes of formulas. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.) (Abstract shortened with permission of school.)