Discovering Chinese Words from Unsegmented Text
暂无分享,去创建一个
In English written text, words are separated by spaces, but in written Chinese text, there are no such separators between words. (See Figure 1.) Thus, effective information retrieval of Chinese text first requires good word segmentation. In this paper, we investigate an efficient algorithm to discover the words and their occurrence probabilities from a corpus of unsegmented text without using a dictionary. Using the probabilities of the words, word segmentation is done according to the maximum likelihood principle. Comparing the segmentation output by the algorithm with the correct segmentation, recall/precision of 65.65%/71.91% is achieved. If some simple post-processing is performed, recall/precision can be boosted up to 97.72%/91.05%.
[1] J. Ponte. USe: A Retargetable Word Segmentation Procedure for Information Retrieval , 1996 .
[2] Richard Sproat,et al. A statistical method for finding word boundaries in Chinese text , 1990 .
[3] Fredric C. Gey,et al. Chinese text retrieval without using a dictionary , 1997, SIGIR '97.
[4] Wing Shing Wong,et al. Automatic Segmentation and Tagging of Hanzi Text Using a Hybrid Algorithm , 1996, IEA/AIE.