论文信息 - Language Identification in Code-Switching Scenario

Language Identification in Code-Switching Scenario

This paper describes a CRF based token level language identification system entry to Language Identification in CodeSwitched (CS) Data task of CodeSwitch 2014. Our system hinges on using conditional posterior probabilities for the individual codes (words) in code-switched data to solve the language identification task. We also experiment with other linguistically motivated language specific as well as generic features to train the CRF based sequence labeling algorithm achieving reasonable results.

Riyaz Ahmad Bhat | Naman Jain

[1] Timothy Baldwin,et al. Automatic Detection and Language Identification of Multilingual Documents , 2014, TACL.

[2] Thamar Solorio,et al. Overview for the Second Shared Task on Language Identification in Code-Switched Data , 2014, CodeSwitch@EMNLP.

[3] Mauro Cettolo,et al. IRSTLM: an open source toolkit for handling large scale language models , 2008, INTERSPEECH.

[4] Ted E. Dunning,et al. Statistical Identification of Language , 1994 .

[5] Timothy Baldwin,et al. Language Identification: The Long and the Short of the Matter , 2010, NAACL.

[6] Andrew McCallum,et al. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[7] Ben King,et al. Labeling the Languages of Words in Mixed-Language Documents using Weakly Supervised Methods , 2013, NAACL.

[8] Chris Callison-Burch,et al. Arabic Dialect Identification , 2014, CL.

[9] Dong Nguyen,et al. Word Level Language Identification in Online Multilingual Communication , 2013, EMNLP.

[10] Mona T. Diab,et al. Token Level Identification of Linguistic Code Switching , 2012, COLING.

[11] Julia Hirschberg,et al. Overview for the First Shared Task on Language Identification in Code-Switched Data , 2014, CodeSwitch@EMNLP.