On the Strength of Incremental Learning

This paper provides a systematic study of incremental learning from noise-free and from noisy data, thereby distinguishing between learning from only positive data and from both positive and negative data. Our study relies on the notion of noisy data introduced in [22]. The basic scenario, named iterative learning, is as follows. In every learning stage, an algorithmic learner takes as input one element of an information sequence for a target concept and its previously made hypothesis and outputs a new hypothesis. The sequence of hypotheses has to converge to a hypothesis describing the target concept correctly. We study the following refinements of this scenario. Bounded example-memory inference generalizes iterative inference by allowing an iterative learner to additionally store an a priori bounded number of carefully chosen data elements, while feedback learning generalizes it by allowing the iterative learner to additionally ask whether or not a particular data element did already appear in the data seen so far. For the case of learning from noise-free data, we show that, where both positive and negative data are available, restrictions on the accessibility of the input data do not limit the learning capabilities if and only if the relevant iterative learners are allowed to query the history of the learning process or to store at least one carefully selected data element. This insight nicely contrasts the fact that, in case only positive data are available, restrictions on the accessibility of the input data seriously affect the capabilities of all types of incremental learning (cf. [18]). For the case of learning from noisy data, we present characterizations of all kinds of incremental learning in terms being independent from learning theory. The relevant conditions are purely structural ones. Surprisingly, where learning from only noisy positive data and from both noisy positive and negative data, iterative learners are already exactly as powerful as unconstrained learning devices.

[1]  John Case,et al.  Vacillatory and BC learning on noisy data , 1996, Theor. Comput. Sci..

[2]  Sanjay Jain,et al.  Open Problems in "Systems That Learn" , 1994, J. Comput. Syst. Sci..

[3]  John Case,et al.  Incremental Concept Learning for Bounded Data Mining , 1997, Inf. Comput..

[4]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[5]  John Case,et al.  Synthesizing Learners Tolerating Computable Noisy Data , 1998, ALT.

[6]  Rokia Missaoui,et al.  An Incremental Concept Formation Approach for Learning from Databases , 1994, Theor. Comput. Sci..

[7]  Thomas Zeugmann,et al.  Incremental Learning from Positive Data , 1996, J. Comput. Syst. Sci..

[8]  Thomas Zeugmann,et al.  Learning Recursive Languages with Bounded Mind Changes , 1993, Int. J. Found. Comput. Sci..

[9]  Thomas Zeugmann,et al.  Language learning in dependence on the space of hypotheses , 1993, COLT '93.

[10]  Rolf Wiehagen.  Limes-Erkennung rekursiver Funktionen durch spezielle Strategien , 1975, J. Inf. Process. Cybern..

[11]  Frank Stephan,et al.  Language Learning from Texts: Mind Changes, Limited Memory and Monotonicity (Extended Abstract). , 1995, COLT 1995.

[12]  Antoine Cornu.  Getting Order Independence in Incremental Learning , 1993 .

[13]  J. Feldman,et al.  Learning Automata from Ordered Examples , 1991 .

[14]  Dana Angluin,et al.  Inductive Inference of Formal Languages from Positive Data , 1980, Inf. Control..

[15]  Jeffrey D. Ullman,et al.  Formal languages and their relation to automata , 1969, Addison-Wesley series in computer science and information processing.

[16]  Antoine Cornuéjols,et al.  Getting Order Independence in Incremental Learning , 1993, ECML.

[17]  Sanjay Jain,et al.  Program Synthesis in the Presence of Infinite Number of Inaccuracies , 1994, J. Comput. Syst. Sci..

[18]  Daniel N. Osherson,et al.  Systems That Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists , 1990 .

[19]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..

[20]  Klaus P. Jantke,et al.  Combining Postulates of Naturalness in Inductive Inference , 1981, J. Inf. Process. Cybern..

[21]  Thomas Zeugmann,et al.  A Guided Tour Across the Boundaries of Learning Recursive Languages , 1995, GOSLER Final Report.

[22]  Ronald L. Rivest,et al.  Learning decision lists , 2004, Machine Learning.

[23]  John Case,et al.  Synthesizing noise-tolerant language learners , 2001, Theor. Comput. Sci..

[24]  Luís Torgo,et al.  Controlled Redundancy in Incremental Rule Learning , 1993, ECML.

[25]  Frank Stephan,et al.  Noisy Inference and Oracles , 1995, Theor. Comput. Sci..