Document Retrieval: Shallow Data, Deep Theories; Historical Reflections, Potential Directions

This paper reviews the development of statistically-based retrieval. Since the 1950s statistical techniques have clearly demonstrated their practical worth and statistical theories their staying power, for document or text retrieval. In the last decade the TREC programme, and the Web, have offered new retrieval challenges to which these methods have successfully risen. They are now one element in the much wider and very productive spread of statistical methods to all areas of information and language processing, in which innovative approaches to modelling their data and tasks are being applied.

[1]  Kui-Lam Kwok,et al.  A network approach to probabilistic information retrieval , 1995, TOIS.

[2]  Stephen E. Robertson,et al.  Probabilistic models of indexing and searching , 1980, SIGIR '80.

[3]  Norbert Fuhr,et al.  Models for retrieval with probabilistic indexing , 1989, Inf. Process. Manag..

[4]  W. Bruce Croft.  A model of cluster searching bases on classification , 1980, Inf. Syst..

[5]  K. Sparck Jones,et al.  KEYWORDS AND CLUMPS , 1964 .

[6]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[7]  Michele Banko,et al.  Headline Generation Based on Statistical Translation , 2000, ACL.

[8]  David Hawking,et al.  Overview of the TREC-2001 Web track , 2002 .

[9]  Norbert Fuhr,et al.  The automatic indexing system AIR/PHYS - from research to applications , 1988, SIGIR '88.

[10]  Steve J. Young,et al.  Speech recognition evaluation: a review of the U.S. CSR and LVCSR programmes , 1998, Comput. Speech Lang..

[11]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[12]  Martin F. Porter,et al.  An algorithm for suffix stripping , 1997, Program.

[13]  Ellen M. Voorhees,et al.  The Sixth Text REtrieval Conference (TREC-6) , 2000, Inf. Process. Manag..

[14]  Nicholas J. Belkin,et al.  Distributed expert problem treatment as a model for information system analysis and design , 1982 .

[15]  Mary Elizabeth Stevens,et al.  Statistical Association Methods for Mechanized Documentation. , 1967 .

[16]  Gerard Salton,et al.  A theory of indexing , 1975, Regional conference series in applied mathematics.

[17]  Nathalie Bely,et al.  Procédures d'analyse sémantique appliquées à la documentation scientifique , 1970 .

[18]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[19]  Gerard Salton,et al.  Another look at automatic text-retrieval systems , 1986, CACM.

[20]  Lynette Hirschman,et al.  MiTAP for Biosecurity: A Case Study , 2002, AI Mag..

[21]  Irma S. Wachtel,et al.  Unit terms in coordinate indexing , 1952 .

[22]  Hans H. Wellisch.  1/71‐1R Sacherschliessung an Bibliotheken bei Einsatz elektronischer Datenverarbeitung. (Subject Retrieval in Libraries with the Aid of EDP.) 1970. Dietrich Roessler. Köln, Greven Verlag. 117 pp. (Arbeiten aus dem Bibliothekar‐Lehrinstitut des Landes Rheinland‐Westfalen, Heft 34.) , 1971 .

[23]  P Grandjean.  [Classification of research]. , 1990, Ugeskrift for laeger.

[24]  Tamas E. Doszkocs,et al.  CITE NLM: natural-language searching in an online catalog , 1983 .

[25]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[26]  Ross Wilkinson,et al.  Chinese Document Retrieval at TREC-6 , 1997, TREC.

[27]  Richard M. Schwartz,et al.  A hidden Markov model information retrieval system , 1999, SIGIR '99.

[28]  Robert N. Oddy,et al.  Information Retrieval Research , 1982 .

[29]  C. J. van Rijsbergen,et al.  A Non-Classical Logic for Information Retrieval , 1997, Comput. J..

[30]  Karen Sparck Jones.  Search term relevance weighting- some recent results , 1979 .

[31]  Donna Harman,et al.  The Second Text Retrieval Conference (TREC-2) , 1995, Inf. Process. Manag..

[32]  John D. Lafferty,et al.  Information retrieval as statistical translation , 1999, SIGIR '99.

[33]  Mary Elizabeth Stevens,et al.  Automatic indexing : a state-of-the art report , 1965 .

[34]  Daniel Marcu,et al.  Summarization beyond sentence extraction: A probabilistic approach to sentence compression , 2002, Artif. Intell..

[35]  Gerard Salton,et al.  On the application of syntactic methodologies in automatic text analysis , 1989, SIGIR '89.

[36]  Gerard Salton,et al.  A Simple Blueprint for Automatic Boolean Query Processing , 1988, Inf. Process. Manag..

[37]  Hans Peter Luhn,et al.  A Statistical Approach to Mechanized Encoding and Searching of Literary Information , 1957, IBM J. Res. Dev..

[38]  Robert L. Mercer,et al.  Class-Based n-gram Models of Natural Language , 1992, CL.

[39]  Stephen E. Robertson,et al.  A probabilistic model of information retrieval: development and comparative experiments - Part 2 , 2000, Inf. Process. Manag..

[40]  Gerard Salton,et al.  A vector space model for automatic indexing , 1975, CACM.

[41]  W. Bruce Croft,et al.  Inference networks for document retrieval , 1989, SIGIR '90.

[42]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[43]  Jimmy J. Lin,et al.  Web question answering: is more always better? , 2002, SIGIR '02.

[44]  Calvin N. Mooers,et al.  Zatocoding applied to mechanical organization of knowledge , 1951 .

[45]  Djoerd Hiemstra,et al.  Term-specific smoothing for the language modeling approach to information retrieval: the importance of a query term , 2002, SIGIR '02.

[46]  W. Bruce Croft,et al.  The use of phrases and structured queries in information retrieval , 1991, SIGIR '91.

[47]  K. Sparck Jones,et al.  What makes an automatic keyword classification effective , 1971 .