论文信息 - A comparison of morpheme and word based document retrieval for Asian languages

A comparison of morpheme and word based document retrieval for Asian languages

Most document retrieval systems are word based. Words are very convenient retrieval units in English but not so in some Asian languages. The task of determining which morphemes constitute words in Vietnamese and Chinese is problematic, and has been assumed to be the reason that word based retrieval does not work so well. The paper examines a number of segmentation algorithms, and then reports on some experiments comparing morpheme and word based retrieval. It shows that morpheme based retrieval is hard to improve on.

Ross Wilkinson | Phil Vines | Van Be Hai Nguyen

[1] John C. Olney,et al. A new technique for detecting patterns of term usage in text corpora , 1976, Inf. Process. Manag..

[2] Gerard Salton,et al. Another look at automatic text-retrieval systems , 1986, CACM.

[3] W. Bruce Croft,et al. A comparison of indexing techniques for Japanese text retrieval , 1993, SIGIR.

[4] Richard Sproat,et al. A statistical method for finding word boundaries in Chinese text , 1990 .

[5] Gerard Salton,et al. On the application of syntactic methodologies in automatic text analysis , 1989, SIGIR '89.

[6] Ian H. Witten,et al. Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[7] Martin Dillon,et al. FASIT: A fully automatic syntactically based indexing system , 1983, J. Am. Soc. Inf. Sci..

[8] Donna Harman,et al. How effective is suffixing , 1991 .

[9] Gwyneth Tseng,et al. Chinese text segmentation for text retrieval: achievements and problems , 1993 .

[10] Paul H. Klingbiel. A technique for machine-aided indexing , 1973, Inf. Storage Retr..