论文信息 - CRFs-Based Chinese Word Segmentation for Micro-Blog with Small-Scale Data

CRFs-Based Chinese Word Segmentation for Micro-Blog with Small-Scale Data

In this paper, we proposed a Chinese word segmentation model for micro-blog text. Although Conditional Random Fields (CRFs) models have been presented to deal with word segmentation, this is still the first time to apply it for the segmentation in the domain of Chinese micro-blog. Different from the genres of common articles, micro-blog has gradually become a new literary with the development of Internet. However, the unavailable of microblog training data has been the obstacle to develop a good segmenter based on trainable models. Considering the linguistic characteristics of the text, we proposed some methods to make the CRFs models suitable for segmentation in the domain of micro-blog. Several experiments have been conducted with different settings and then an optimal tagging method and feature templates have been designed. The proposed model has been implemented for the Second CIPS-SIGHAN Joint Conference on Chinese Language Processing Bakeoff (Bakeoff-2012) and achieves a very high Fmeasure of 93.38% within the test set of 5,000 micro-blog sentences. One of our main contri

[1] Ming Chui Dong,et al. CSAT: A Chinese Segmentation and Tagging Module Based on the Interpolated Probabilistic Model , 2006 .

[2] Andrew McCallum,et al. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[3] Hwee Tou Ng,et al. A Maximum Entropy Approach to Chinese Word Segmentation , 2005, SIGHAN@IJCNLP 2005.

[4] Adwait Ratnaparkhi,et al. A Maximum Entropy Model for Part-Of-Speech Tagging , 1996, EMNLP.

[5] Shiwen Yu,et al. Specification for Corpus Processing at Peking University: Word Segmentation, POS Tagging and Phonetic Notation , 2003, J. Chin. Lang. Comput..

[6] Nianwen Xu,et al. Chinese Word Segmentation as Character Tagging , 2003, Int. J. Comput. Linguistics Chin. Lang. Process..

[7] Hai Zhao,et al. Unsupervised Segmentation Helps Supervised Learning of Character Tagging for Word Segmentation and Named Entity Recognition , 2008, IJCNLP.

[8] Hai Zhao,et al. An Improved Chinese Word Segmentation System with Conditional Random Field , 2006, SIGHAN@COLING/ACL.

[9] Zhao Hai,et al. Chinese Word Segmentation: A Decade Review , 2007 .

[10] Xiaojun Wan,et al. CRF-based Experiments for Cross-Domain Chinese Word Segmentation at CIPS-SIGHAN-2010 , 2010, CIPS-SIGHAN.

[11] Daniel Jurafsky,et al. A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005 , 2005, IJCNLP.