FASTR : A Unification-Based Front-End to Automatic Indexing

Most natural language processing approaches to full-text information retrieval are based on indexing documents by the occurrences of controlled terms they contain. An important problem with this approach is that terms accept numerous variations, and can therefore cause many documents not to be retrieved although being relevant. For example, "myeloid leukaemia cells" and "myeloid and erythoid cell" are two occurrences of "myeloid cell" which cannot be detected without an account of local morpho-syntactic variations. In this paper, we present a linguistic analysis of the observed variations and a three-tier constraint-based formalism for representing them. This technique has been implemented and results in FASTR, a natural language processing tool that extracts terms and their variants from full-text documents. We justify the choice of a unification-based formalism by its expressivity and by the addition of conceptual and computational devices which make the parser computationally tractable. Contrary to the generally accepted idea, high quality natural language processing through unification and industrial requirements can fit together, provided that the application is carefully designed in order to control and minimize data accesses and computation times. The effectiveness of FASTR for extracting correct occurrences is supported by experiments on two English corpora of scientific abstracts and a list of 71,623 controlled terms. We report that an account of three kinds of variants (insertions, permutations and coordinations) increases recall by 16.7% without altering precision.