Automatic text structuring and retrieval-experiments in automatic encyclopedia searching

Many conventional approaches to text analysis and information retrieval prove ineffective when large text collections must be processed in heterogeneous subject areas. An alternative text manipulation system is outlined useful for the retrieval of large heterogeneous texts, and for the recognition of content similarities between text excerpts, based on flexible text matching procedures carried out in several contexts of different scope. The methods are illustrated by search experiments performed with the 29-volume Funk and Wagnalls encyclopedia. Approaches to Text Retrieval We are addressing the problem of text retrieval from large, heterogeneous text databases, where the vocabulary varies widely and the subject matter is unrestricted. This includes newspaper articles, newswire dispatches, textbooks, dictionaries and encyclopedias, manuals, magazine articles, and so on. Necessarily, the retrieval operations must be preceded by some form of text analysis, or text indexing operation, leading to the assignment to query and document texts of appropriate content identifications. When content identifiers are attached to text items, the text retrieval operations can be carried out by comparing the identifications assigned to the various text items with the query identifications, and retrieving texts that appear sufficiently similar to the corresponding queries. [1-2] ● Department of Computer Science, Cornell University, Ithaca, NY 14853-7501. This study was supported in part by the National Science Foundation under grant IRI 89-15847. Permission to copy without fee all or part of this material ia granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice ie given that copying is by permieaion of the Association for Computing Machinery. To copy otherwiee, or to republish, requiree a fee and/or specific permission. @ 1991 ACM 0.89791.448-1/91/0009/0021 ,.,$1.50 In discussing the text analysis operation, it is customary to contrast the so-called keyword approach with more sophisticated procedures based in part on linguistic and logical considerations. In the keyword approach, single-term concepts, known as keywords,