Identification in the Limit of Systematic-Noisy Languages

To study the problem of learning from noisy data, the common approach is to use a statistical model of noise. The influence of the noise is then considered according to pragmatic or statistical criteria, by using a paradigm taking into account a distribution of the data. In this article, we study the noise as a nonstatistical phenomenon, by defining the concept of systematic noise. We establish various ways of learning (in the limit) from noisy data. The first is based on a technique of reduction between problems and consists in learning from the data which one knows noisy, then in denoising the learned function. The second consists in denoising on the fly the training examples, thus to identify in the limit good examples, and then to learn from noncorrupted data. We give in both cases sufficient conditions so that learning is possible and we show through various examples (coming in particular from the field of the grammatical inference) that our techniques are complementary.

[1]  Barak A. Pearlmutter,et al.  Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm , 1998, ICGI.

[2]  Yuji Takada.  Grammatical Interface for Even Linear Languages Based on Control Sets , 1988, Inf. Process. Lett..

[3]  E. Mark Gold,et al.  Complexity of Automaton Identification from Given Data , 1978, Inf. Control..

[4]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[5]  Satoshi Kobayashi,et al.  On Approximately Identifying Concept Classes in the Limit , 1995, ALT.

[6]  José Oncina,et al.  Learning Stochastic Regular Grammars by Means of a State Merging Method , 1994, ICGI.

[7]  Leslie G. Valiant,et al.  Cryptographic Limitations on Learning Boolean Formulae and Finite Automata , 1993, Machine Learning: From Theory to Applications.

[8]  T. Ho,et al.  Data Complexity in Pattern Recognition , 2006 .

[9]  John Case,et al.  Synthesizing noise-tolerant language learners , 2001, Theor. Comput. Sci..

[10]  Michael J. Fischer,et al.  The String-to-String Correction Problem , 1974, JACM.

[11]  R. M. Wharton.  Approximate Language Identification , 1974, Inf. Control..

[12]  Takashi Yokomori,et al.  Inductive Learning of Regular Sets from Examples: A Rough Set Approach , 1994 .

[13]  Naoki Abe,et al.  On the computational complexity of approximating distributions by probabilistic automata , 1990, Machine Learning.

[14]  Laurent Miclet,et al.  Définitions et premières expériences en apprentissage par analogie dans les séquences , 2005, CAP.

[15]  Ah Chung Tsoi,et al.  Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference , 2001, Machine Learning.

[16]  Francisco Casacuberta,et al.  Topology of Strings: Median String is NP-Complete , 1999, Theor. Comput. Sci..

[17]  Zdzislaw Pawlak,et al.  Theory of Rough Sets: A New Methodology for Knowledge Discovery (Abstract) , 1990, International Conference on Computing and Information.

[18]  Marc Sebban,et al.  On State Merging in Grammatical Inference: A Statistical Approach for Dealing with Noisy Data , 2003, ICML.

[19]  Colin de la Higuera,et al.  A bibliographical study of grammatical inference , 2005, Pattern Recognit..

[20]  Colin de la Higuera,et al.  Data Complexity Issues in Grammatical Inference , 2006 .

[21]  Rafael C. Carrasco,et al.  Grammatical Inference and Applications , 1994, Lecture Notes in Computer Science.

[22]  Yasubumi Sakakibara,et al.  Recent Advances of Grammatical Inference , 1997, Theor. Comput. Sci..

[23]  C. D. L. Higuera.  Complexity and Reductions Issues in Grammatical Inference , 2005 .

[24]  Marc Sebban,et al.  Improvement of the State Merging Rule on Noisy Data in Probabilistic Grammatical Inference , 2003, ECML.

[25]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[26]  Francisco Casacuberta,et al.  Probabilistic finite-state machines - part I , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Daniel Fredouille,et al.  Unambiguous Automata Inference by Means of State-Merging Methods , 2003, ECML.

[28]  Hendrik Blockeel,et al.  Machine Learning: ECML 2003 , 2003, Lecture Notes in Computer Science.

[29]  Frank Stephan,et al.  Noisy Inference and Oracles , 1995, Theor. Comput. Sci..

[30]  Colin de la Higuera.  Data complexity in grammatical inference , 2006 .

[31]  Alexander Clark,et al.  PAC-learnability of Probabilistic Deterministic Finite State Automata , 2004, J. Mach. Learn. Res..

[32]  E. Mark Gold,et al.  Language Identification in the Limit , 1967, Inf. Control..