The Exploration and Application of K-medoids in Text Clustering

Clustering algorithms is a statistical analysis method for classifying samples/indexes. The traditional text clustering algorithm is complicated and not convenient for data processing. Therefore, we proposed a new text clustering algorithm based on K-medoids. The new text clustering algorithm combines document category with semantics contribution. The new clustering algorithm can not only optimize the document frequency, but also take consideration of influence of the document category on the characteristic weight. The new text clustering algorithm was shown as follows: first, combine the proposed semantic contribution with fuzzy cluster, and vested the document (with no category information) category thereby; then we proposed the category information entropy and combined it with the semantic contribution in order to modify the traditional TF-IDF weight calculation method. We found the new text clustering algorithm was superior to the traditional weight calculation method after testing it in open platform of Chinese text categorization corpus data set. Therefore, we concluded that the new text clustering algorithm might have vast foreground of application. To solve the shortcomings of the traditional weight calculation method of feature items, text clustering algorithm based on K-medoids was proposed. The frequency and inverse document frequency were improved, and the influence of document category on feature weight was further studied. At the same time, because there may not be any standard classification datasets in practice, a new weight calculation method combining category and semantic contribution was proposed. First, the semantic contribution was proposed and then combined with fuzzy clustering. A text set with category information was obtained by rough clustering of text set without category information. Then, the category information entropy was proposed and combined with the semantic contribution to improve the traditional TF-IDF weight calculation method. Thus, a more effective weight calculation method was obtained. The Chinese text categorization corpus dataset in open platform of Chinese natural language processing of Fudan University was used for testing. The results showed that the new method for weight calculation of feature items was superior to the traditional weight calculation method. It is concluded that the improved text clustering algorithm can be used in a wider range of occasions.