Finding Topics in News Web Pages by Parameter-free Clustering

Abstract Topic detection is a novel technology which structures news stories into several topics. Present topic detection approaches are mainly based on clustering algorithms such as single pass or agglomerative clus-tering, and all these algorithms need at least one input parameter. We proposed a novel clustering algorithm which auto-matically determines the parameters for each corpus. Experimental results show that the parameters derived are close to optimal, and our algorithm has similar accuracy as the UPGMA algorithm which is manually set with optimal parameters. Another advantage of our algorithm is that it runs much faster than the UPGMA algorithm. Keywords : Topic Detection, Clustering Algorithm, Parameter Free, Similarity Distribution 1. Introduction Nowadays web pages have become the fastest ways for us to achieve news and publish individual opinions. It is hard to identify what pages exactly we want. By giving a set of keywords to search engines such as Google and Yahoo!, we can obtain a very long list of URLs referring to Web pages. However, it is still a difficult work to grasp and summarize the contents quickly from the search results. We need a convenient and efficient way to learn “what’s happened” or “what’s hot” from large number of news web pages. Topic detection is such a method which automatically finds topics in a group of news corpuses. At present, topic detection is widely applied to web information organization, such as automatic construction of online news issues [1] or organization of RSS news from different sources on topics [2]. It is easy to think of the use of data mining algorithms to find topics. Actually, clustering is the core algorithm of present topic detection approaches. Common clustering algorithms used for topic detection include single pass, kNN and agglomerative clustering [1][3][4][5]. A drawback of these algorithms is that they can not run automatically without manual intervention. A common solution for the above problem is to use fixed parameters for all corpuses when they are similar in size and structure. A representative work on this approach is [1] which uses the UPGMA agglomerative clustering algorithm and specifies a fixed threshold to control the termination of the algorithm. But according to our experiments, when size or structure varies among corpuses, the optimal thresholds vary largely. We suppose that different corpuses have different optimal parameters regardless of which algorithm and parameters are used, and it is impossible for users to determine the optimal