论文信息 - A Comparison of Language Identification Approaches on Short, Query-Style Texts

A Comparison of Language Identification Approaches on Short, Query-Style Texts

In a multi-language Information Retrieval setting, the knowledge about the language of a user query is important for further processing. Hence, we compare the performance of some typical approaches for language detection on very short, query-style texts. The results show that already for single words an accuracy of more than 80% can be achieved, for slightly longer texts we even observed accuracy values close to 100%.

Thomas Gottron | Nedim Lipka

[1] W. B. Cavnar,et al. N-gram-based text categorization , 1994 .

[2] Yan Xu,et al. A Search Engine Based on Query Logs, and Search Log Analysis by Automatic Language Identification , 2009, CLEF.

[3] Yan Xu,et al. A Search Engine based on Query Logs and Search Log Analysis at the University of Sunderland , 2009, CLEF.

[4] Radim Rehurek,et al. Language Identification on the Web: Extending the Dictionary Method , 2009, CICLing.

[5] Ted E. Dunning,et al. Statistical Identification of Language , 1994 .

[6] Mária Bieliková,et al. Comparing Natural Language Identification Methods based on Markov Processes ? , 2007 .

[7] Etienne Barnard,et al. Analysis of phoneme-based features for language identification , 1994, Proceedings of ICASSP '94. IEEE International Conference on Acoustics, Speech and Signal Processing.

[8] Yiming Yang,et al. RCV1: A New Benchmark Collection for Text Categorization Research , 2004, J. Mach. Learn. Res..

[9] William John Teahan,et al. Text classification and segmentation using minimum cross-entropy , 2000, RIAO.

[10] Ching Y. Suen,et al. n-Gram Statistics for Natural Language Understanding and Text Processing , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.