论文信息 - A Phonetic-Based Approach to Chinese Chat Text Normalization

A Phonetic-Based Approach to Chinese Chat Text Normalization

Chatting is a popular communication media on the Internet via ICQ, chat rooms, etc. Chat language is different from natural language due to its anomalous and dynamic natures, which renders conventional NLP tools inapplicable. The dynamic problem is enormously troublesome because it makes static chat language corpus outdated quickly in representing contemporary chat language. To address the dynamic problem, we propose the phonetic mapping models to present mappings between chat terms and standard words via phonetic transcription, i.e. Chinese Pinyin in our case. Different from character mappings, the phonetic mappings can be constructed from available standard Chinese corpus. To perform the task of dynamic chat language term normalization, we extend the source channel model by incorporating the phonetic mapping models. Experimental results show that this method is effective and stable in normalizing dynamic chat language terms.

Kam-Fai Wong | Yunqing Xia | Wenjie Li

[1] Anita Pincas,et al. Report into the use of Chat in education , 2006 .

[2] Sergei Nirenburg,et al. A Statistical Approach to Machine Translation , 2003 .

[3] Frankie James,et al. Modified Kneser-Ney Smoothing of n-gram Models , 2000 .

[4] Wei Gao,et al. NIL Is Not Nothing: Recognition of Chinese Network Informal Language Expressions , 2005, IJCNLP.

[5] Slava M. Katz,et al. Estimation of probabilities from sparse data for the language model component of a speech recognizer , 1987, IEEE Trans. Acoust. Speech Signal Process..

[6] John Cocke,et al. A Statistical Approach to Machine Translation , 1990, CL.

[7] Yuan Bao-zong. An Kind of Chinese Text Strings' Similarity and its Application in Speech Recognition , 2003 .

[8] Kam-Fai Wong,et al. Constructing A Chinese Chat Language Corpus with A Two-Stage Incremental Annotation Approach , 2006, LREC.

[9] Kam-Fai Wong,et al. Anomaly Detecting within Dynamic Chinese Chat Text , 2006, Workshop On New Text Wikis And Blogs And Other Dynamic Text Sources.