THESUS: Organizing Web document collections based on link semantics

Abstract.The requirements for effective search and management of the WWW are stronger than ever. Currently Web documents are classified based on their content not taking into account the fact that these documents are connected to each other by links. We claim that a page’s classification is enriched by the detection of its incoming links’ semantics. This would enable effective browsing and enhance the validity of search results in the WWW context. Another aspect that is underaddressed and strictly related to the tasks of browsing and searching is the similarity of documents at the semantic level. The above observations lead us to the adoption of a hierarchy of concepts (ontology) and a thesaurus to exploit links and provide a better characterization of Web documents. The enhancement of document characterization makes operations such as clustering and labeling very interesting. To this end, we devised a system called THESUS. The system deals with an initial sets of Web documents, extracts keywords from all pages’ incoming links, and converts them to semantics by mapping them to a domain’s ontology. Then a clustering algorithm is applied to discover groups of Web documents. The effectiveness of the clustering process is based on the use of a novel similarity measure between documents characterized by sets of terms. Web documents are organized into thematic subsets based on their semantics. The subsets are then labeled, thereby enabling easier management (browsing, searching, querying) of the Web. In this article, we detail the process of this system and give an experimental analysis of its results.

[1]  Oren Etzioni,et al.  Web document clustering: a feasibility demonstration , 1998, SIGIR '98.

[2]  Michalis Vazirgiannis,et al.  c ○ 2001 Kluwer Academic Publishers. Manufactured in The Netherlands. On Clustering Validation Techniques , 2022 .

[3]  Prabhakar Raghavan,et al.  Mining the Link Structure of the World Wide Web , 1998 .

[4]  Eli Upfal,et al.  Web search using automatic classification , 1996, WWW 1996.

[5]  Philip Resnik,et al.  Using Information Content to Evaluate Semantic Similarity in a Taxonomy , 1995, IJCAI.

[6]  Jon M. Kleinberg,et al.  Automatic Resource Compilation by Analyzing Hyperlink Structure and Associated Text , 1998, Comput. Networks.

[7]  Philip S. Yu,et al.  On the merits of building categorization systems by supervised clustering , 1999, KDD '99.

[8]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[9]  Chinatsu Aone,et al.  Fast and effective text mining using linear-time document clustering , 1999, KDD '99.

[10]  Douglas H. Fisher,et al.  Knowledge Acquisition Via Incremental Conceptual Clustering , 1987, Machine Learning.

[11]  Isabel F. Cruz,et al.  The Emerging Semantic Web , 2002 .

[12]  B. Nguyen,et al.  Organizing Web Documents into Thematic Subsets Using an Ontology , 2003 .

[13]  Susan T. Dumais,et al.  Hierarchical classification of Web content , 2000, SIGIR '00.

[14]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[15]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[16]  Iraklis Varlamis,et al.  THESUS: Organizing Web Doc-ument Collections Based On Semantics And Clustering , 2002 .

[17]  Hans-Peter Frei,et al.  Concept based query expansion , 1993, SIGIR.

[18]  Dan Klein,et al.  Evaluating strategies for similarity search on the web , 2002, WWW '02.

[19]  HalkidiMaria,et al.  THESUS: Organizing Web document collections based on link semantics , 2003, VLDB 2003.

[20]  Christine Jacquin,et al.  Indexing a web site with a terminology oriented ontology , 2001, SWWS.

[21]  Iraklis Varlamis,et al.  Web document searching using enhanced hyperlink semantics based on XML , 2001, Proceedings 2001 International Database Engineering and Applications Symposium.

[22]  Ewa Orlowska,et al.  A Rough Set Model of Information Retrieval , 1996, Fundam. Informaticae.

[23]  John Murphy,et al.  Using WordNet as a Knowledge Base for Measuring Semantic Similarity between Words , 1994 .

[24]  Hans-Peter Kriegel,et al.  Incremental Clustering for Mining in a Data Warehousing Environment , 1998, VLDB.

[25]  David M. Pennock,et al.  Using web structure for classifying and describing web pages , 2002, WWW.

[26]  N. Guarino,et al.  Formal Ontology in Information Systems : Proceedings of the First International Conference(FOIS'98), June 6-8, Trento, Italy , 1998 .

[27]  Víctor Pàmies,et al.  Open Directory Project , 2003 .

[28]  Dimitrios Gunopulos,et al.  Efficient and tumble similar set retrieval , 2001, SIGMOD '01.

[29]  Nicola Guarino,et al.  Formal Ontology and Information Systems , 1998 .

[30]  Philip Resnik,et al.  Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language , 1999, J. Artif. Intell. Res..

[31]  Heikki Mannila,et al.  Distance measures for point sets and their computation , 1997, Acta Informatica.

[32]  Christiane Fellbaum,et al.  Book Reviews: WordNet: An Electronic Lexical Database , 1999, CL.

[33]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[34]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[35]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[36]  Robert Wilensky,et al.  Robust Hyperlinks Cost Just Five Words Each , 2000 .

[37]  Martha Palmer,et al.  Verb Semantics and Lexical Selection , 1994, ACL.

[38]  Jon M. Kleinberg,et al.  Mining the Web's Link Structure , 1999, Computer.