论文信息 - SoMaJo: State-of-the-art tokenization for German web and social media texts

SoMaJo: State-of-the-art tokenization for German web and social media texts

In this paper we describe SoMaJo, a rulebased tokenizer for German web and social media texts that was the best-performing system in the EmpiriST 2015 shared task with an average F1-score of 99.57. We give an overview of the system and the phenomena its rules cover, as well as a detailed error analysis. The tokenizer is available as free software.

Thomas Proisl | Peter Uhrig | Thomas Proisl | P. Uhrig

[1] Helmut Schmidt,et al. Probabilistic part-of-speech tagging using decision trees , 1994 .

[2] Helmut Schmid,et al. Improvements in Part-of-Speech Tagging with an Application to German , 1999 .

[3] Dan Klein,et al. Accurate Unlexicalized Parsing , 2003, ACL.

[4] Julian M. Pine,et al. Constructing a Language: A Usage-Based Theory of Language Acquisition. , 2004 .

[5] Jan Goyvaerts,et al. Regular Expressions Cookbook , 2009 .

[6] Stefan Evert,et al. KLUE: Simple and robust methods for polarity classification , 2013, *SEMEVAL.

[7] Bryan Jurish,et al. Word and Sentence Tokenization with Hidden Markov Models , 2013, J. Lang. Technol. Comput. Linguistics.

[8] Stefan Evert,et al. SentiKLUE: Updating a Polarity Classifier in 48 Hours , 2014, *SEMEVAL.