Document and passage retrieval based on hidden Markov models

Introduced is a new approach to Information Retrieval developed on the basis of Hidden Markov Models (HMMs). HMMs are shown to provide a mathematically sound framework for retrieving documenta—documents with predefined boundaries and also entities of information that are of arbitrary lengths and formats (passage retrieval). Our retrieval model is shown to encompass promising capabilities: First, the position of occurrences of indexing features can be used for indexing. Positional information is essential, for instance, when considering phrases, negation, and the proximity of features. Second, from training collections we can derive automatically optimal weights for arbitrary features. Third, a query dependent structure can be determined for every document by segmenting the documents into passages that are either relevant or irrelevant to the query. The theoretical analysis of our retrieval model is complemented by the results of preliminary experiments.

[1]  Richard Rose,et al.  A hidden Markov model based keyword recognition system , 1990, International Conference on Acoustics, Speech, and Signal Processing.

[2]  Ying He,et al.  Handwritten word recognition using HMM with adaptive length Viterbi algorithm , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[3]  Norbert Fuhr,et al.  Probabilistic Models in Information Retrieval , 1992, Comput. J..

[4]  Julian M. Kupiec,et al.  Robust part-of-speech tagging using a hidden Markov model , 1992 .

[5]  Rainer Kuhlen,et al.  Passagen-Retrieval - auch eine Möglichkeit der automatischen Verknüpfung in Hypertexten , 1993, Information Retrieval.

[6]  C. J. van Rijsbergen,et al.  Information Retrieval , 1979, Encyclopedia of GIS.

[7]  Biing-Hwang Juang,et al.  Hidden Markov Models for Speech Recognition , 1991 .

[8]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[9]  J. Kupiec.  Hidden Markov estimation for unrestricted stochastic context-free grammars , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[10]  Alistair Moffat,et al.  Retrieval of Partial Documents , 1993, TREC.

[11]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[12]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[13]  John O'Connor,et al.  Retrieval of answer-sentences and answer-figures from papers by text searching , 1975, Inf. Process. Manag..

[14]  James Allan,et al.  Approaches to passage retrieval in full text information systems , 1993, SIGIR.

[15]  Jonathan J. Hull,et al.  A hidden Markov model for language syntax in text recognition , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[16]  Udo Hahn,et al.  Topic parsing: Accounting for text macro structures in full-text analysis , 1990, Inf. Process. Manag..

[17]  David L. Waltz,et al.  Statistical methods, artificial intelligence, and information retrieval , 1992 .

[18]  Lynn Wilcox,et al.  HMM-based wordspotting for voice editing and indexing , 1991, EUROSPEECH.

[19]  R. Adams.  Proceedings , 1947 .

[20]  Christian Plaunt,et al.  Subtopic structuring for full-length document access , 1993, SIGIR.

[21]  John O'Connor,et al.  Answer-passage retrieval by text searching , 1980, J. Am. Soc. Inf. Sci..

[22]  L. R. Rabiner,et al.  An introduction to the application of the theory of probabilistic functions of a Markov process to automatic speech recognition , 1983, The Bell System Technical Journal.

[23]  Peter Willett,et al.  Searching for historical word-forms in a database of 17th-century English text using spelling-correction methods , 1992, SIGIR '92.