Error-tolerant Finite-state Recognition with Applications to Morphological Analysis and Spelling Correction

This paper presents the notion of error-tolerant recognition with finite-state recognizers along with results from some applications. Error-tolerant recognition enables the recognition of strings that deviate mildly from any string in the regular set recognized by the underlying finite-state recognizer. Such recognition has applications to error-tolerant morphological processing, spelling correction, and approximate string matching in information retrieval. After a description of the concepts and algorithms involved, we give examples from two applications: in the context of morphological analysis, error-tolerant recognition allows misspelled input word forms to be corrected and morphologically analyzed concurrently. We present an application of this to error-tolerant analysis of the agglutinative morphology of Turkish words. The algorithm can be applied to morphological analysis of any language whose morphology has been fully captured by a single (and possibly very large) finite-state transducer, regardless of the word formation processes and morphographemic phenomena involved. In the context of spelling correction, error-tolerant recognition can be used to enumerate candidate correct forms from a given misspelled string within a certain edit distance. Error-tolerant recognition can be applied to spelling correction for any language, if (a) it has a word list comprising all inflected forms, or (b) its morphology has been fully described by a finite-state transducer. We present experimental results for spelling correction for a number of languages. These results indicate that such recognition works very efficiently for candidate generation in spelling correction for many European languages (English, Dutch, French, German, and Italian, among others) with very large word lists of root and inflected forms (some containing well over 200,000 forms), generating all candidate solutions within 10 to 45 milliseconds (with an edit distance of 1) on a SPARCStation 10/41. For spelling correction in Turkish, error-tolerant recognition operating with a (circular) recognizer of Turkish words (with about 29,000 states and 119,000 transitions) can generate all candidate words in less than 20 milliseconds, with an edit distance of 1.

[1]  Yves Schabes,et al.  Deterministic Part-of-Speech Tagging with Finite-State Transducers , 1995, Comput. Linguistics.

[2]  Xabier Arregi,et al.  A Morphological Analysis Based Method for Spelling Correction , 1993, EACL.

[3]  Kemal Oflazer,et al.  Two-level Description of Turkish Morphology , 1993, EACL.

[4]  Chris Mellish,et al.  Natural Language Processing in Pop-11: An Introduction to Computational Linguistics , 1989 .

[5]  Lauri Karttunen,et al.  Two-Level Morphology with Composition , 1992, COLING.

[6]  M.McGee Wood,et al.  Natural language processing in Prolog , 1990 .

[7]  Kemal Oflazer,et al.  Spelling Correction in Agglutinative Languages , 1994, ANLP.

[8]  Jorge Hankamer,et al.  Morphological parsing and the lexicon , 1989 .

[9]  Steven Krauwer,et al.  Proceedings of the sixth conference on European chapter of the Association for Computational Linguistics , 1993 .

[10]  Gerald Gazdar,et al.  Natural Language Processing in PROLOG: An Introduction to Computational Linguistics , 1989 .

[11]  Lauri Karttunen,et al.  Two-level rule compiler , 1992 .

[12]  Richard Sproat,et al.  Morphology and computation , 1992 .

[13]  George Anton Kiraz,et al.  A Morphographemic Model for Error Correction in Nonconcatenative Strings , 1995, ACL.

[14]  Richard Sproat,et al.  Review of PC-KIMMO: a two-level processor for morphological analysis by Evan L. Antworth. Summer Institute of Linguistics 1990 , 1991 .

[15]  E. Myers,et al.  Approximate matching of regular expressions. , 1989, Bulletin of mathematical biology.

[16]  R. J. Nelson,et al.  Introduction to Automata , 1968 .

[17]  Moti Schneider,et al.  The utilization of fuzzy sets in the recognition of imperfect strings , 1992 .

[18]  Lauri Karttunen.  Constructing Lexical Transducers , 1994, COLING.

[19]  Jeffrey D. Ullman,et al.  Introduction to Automata Theory, Languages and Computation , 1979 .

[20]  Fred J. Damerau,et al.  A technique for computer detection and correction of spelling errors , 1964, CACM.

[21]  Atro Voutilainen,et al.  Ambiguity resolution in a reductionistic parser , 1993, EACL.

[22]  Jean Véronis.  Morphosyntactic correction in natural language interfaces , 1988, COLING.

[23]  Karen Kukich,et al.  Techniques for automatically correcting words in text , 1992, CSUR.

[24]  Kemal Oflazer,et al.  Tagging and Morphological Disambiguation of Turkish Text , 1994, ANLP.