On a generalized notion of mistake bounds

This paper proposes the use of constructive ordinals as mistake bounds in the on-line learning model. This approach elegantly generalizes the applicability of the on-line mistake bound model to learnability analysis of very expressive concept classes like pattern languages, unions of pattern languages, elementary formal systems, and minimal models of logic programs. The main result in the paper shows that the topological property of effective finite bounded thickness is a sufficient condition for on-line learnability with a certain ordinal mistake bound. An interesting characterization of the on-line learning model is shown in terms of the identification in the limit framework. It is established that the classes of languages learnable in the on-line model with a mistake bound of α are exactly the same as the classes of languages learnable in the limit from both positive and negative data by a Popperian, consistent learner with a mind change bound of α. This result nicely builds a bridge between the two models. C © 2001 Academic Press

[1]  Keith Wright.  Identification of unions of languages drawn from an identifiable class , 1989, COLT '89.

[2]  Arun Sharma,et al.  Elementary formal systems, intrinsic complexity, and procrastination , 1997, COLT '96.

[3]  Dana Angluin,et al.  Inductive Inference of Formal Languages from Positive Data , 1980, Inf. Control..

[4]  Rolf Wiehagen,et al.  Learning and Consistency , 1995, GOSLER Final Report.

[5]  M. R. K. Krishna Rao.  A Class of Prolog Programs Inferable from Positive Data , 1996, ALT.

[6]  John Case,et al.  Not-So-Nearly-Minimal-Size Program Inference , 1995, GOSLER Final Report.

[7]  Arun Sharma,et al.  Generalized notions of mind change complexity , 1997, COLT '97.

[8]  Jr. Hartley Rogers.  Theory of Recursive Functions and Effective Computability , 1969 .

[9]  Carl H. Smith,et al.  On the role of procrastination for machine learning , 1992, COLT '92.

[10]  J. W. Lloyd,et al.  Foundations of logic programming; (2nd extended ed.) , 1987 .

[11]  Carl H. Smith,et al.  General Inductive Inference Types Based on Linearly-Ordered Sets , 1996, STACS.

[12]  N. Littlestone.  Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm , 1987, 28th Annual Symposium on Foundations of Computer Science (sfcs 1987).

[13]  Frank Stephan,et al.  Resource bounded next value and explanatory identification: learning automata, patterns and polynomials on-line , 1997, COLT '97.

[14]  Hiroki Arimura,et al.  Inductive Inference of Prolog Programs with Linear Data Dependency from Positive Data , 1993 .

[15]  G. Sacks.  Higher recursion theory , 1990 .

[16]  Dana Angluin,et al.  Finding Patterns Common to a Set of Strings , 1980, J. Comput. Syst. Sci..

[17]  Ayumi Shinohara,et al.  Learning Elementary Formal Systems and an Application to Discovering Motifs in Proteins , 1991 .

[18]  John Case,et al.  Refinements of inductive inference by Popperian and reliable machines , 1994, Kybernetika.

[19]  Carl H. Smith,et al.  On the Role of Procrastination in Machine Learning , 1993, Inf. Comput..

[20]  Andris Ambainis,et al.  Ordinal Mind Change Complexity of Language Identification , 1997, EuroCOLT.

[21]  Paul Young,et al.  An introduction to the general theory of algorithms , 1978 .

[22]  Ehud Shapiro,et al.  Inductive Inference of Theories from Facts , 1991, Computational Logic - Essays in Honor of Alan Robinson.

[23]  Stephen Cole Kleene,et al.  On notation for ordinal numbers , 1938, Journal of Symbolic Logic.

[24]  Andris Ambainis,et al.  Ordinal Mind Change Complexity of Language Identification , 1997, Theor. Comput. Sci..