论文信息 - Rule-Based Normalization of Historical Texts

Rule-Based Normalization of Historical Texts

This paper deals with normalization of language data from Early New High German. We describe an unsupervised, rulebased approach which maps historical wordforms to modern wordforms. Rules are specified in the form of context-aware rewrite rules that apply to sequences of characters. They are derived from two aligned versions of the Luther bible and weighted according to their frequency. The evaluation shows that our approach (83%‐91% exact matches) clearly outperforms the baseline (65%).

Stefanie Dipper | Marcel Bollmann | Florian Petran

[1] Bryan Jurish,et al. More than Words: Using Token Context to Improve Canonicalization of Historical German , 2010, J. Lang. Technol. Comput. Linguistics.

[2] Fabienne Braune,et al. Improved Unsupervised Sentence Alignment for Symmetrical and Asymmetrical Parallel Corpora , 2010, COLING.

[3] Hermann Ney,et al. A Systematic Comparison of Various Statistical Alignment Models , 2003, CL.

[4] Philipp Koehn,et al. Europarl: A Parallel Corpus for Statistical Machine Translation , 2005, MTSUMMIT.

[5] Werner Besch,et al. Handschrift und Druck , 2000 .

[6] Werner Besch,et al. Die Rolle Luthers für die deutsche Sprachgeschichte , 2000 .

[7] Norbert Fuhr,et al. Generating Search Term Variants for Text Collections with Historic Spellings , 2006, ECIR.

[8] Klaus U. Schulz,et al. Unsupervised Learning of Edit Distance Weights for Retrieving Historical Spelling Variations , 2007 .

[9] Dawn Archer,et al. Automatic Standardization of Spelling for Historical Text Mining , 2009 .

[10] W. Heeringa,et al. Predicting intelligibility and perceived linguistic distance by means of the Levenshtein algorithm , 2008 .