Assessing term effectiveness in the interactive information access process

This study addresses the question of whether the way in which sets of query terms are identified has an impact on the effectiveness of users' information seeking efforts. Query terms are text strings used as input to an information access system; they are products of a method or grammar that identifies a set of query terms. We conducted an experiment that compared the effectiveness of sets of query terms identified for a single book by three different methods. One had been previously prepared by a human indexer for a back-of-the-book index. The other two were identified by computer programs that used a combination of linguistic and statistical criteria to extract terms from full text. Effectiveness was measured by (1) whether selected query terms led participants to correct answers and (2) how long it took participants to obtain correct answers. Our results show that two sets of terms - the human terms and the set selected according to the linguistically more sophisticated criteria - were significantly more effective than the third set of terms. This single case demonstrates that query languages do have a measurable impact on the effectiveness of query term languages in the interactive information access process. The procedure described in this paper can be used to assess the effectiveness for information seekers of query terms identified by any query language.

[1]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[2]  Carol Collier Kuhlthau,et al.  A Principle of Uncertainty for Information seeking , 1993, J. Documentation.

[3]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[4]  Slava M. Katz,et al.  Technical terminology: some linguistic properties and an algorithm for identification in text , 1995, Natural Language Engineering.

[5]  William R. Hersh,et al.  A task-oriented approach to information retrieval evaluation , 1996 .

[6]  Diane Kelly,et al.  A model for quantitative evaluation of an end-to-end question-answering system , 2007 .

[7]  Harold Borko,et al.  Automatic indexing , 1981, ACM '81.

[8]  S. Raman,et al.  Natural language interface to video database , 2001, Nat. Lang. Eng..

[9]  Nina Wacholder,et al.  Toward a Task-based Gold Standard for Evaluation of NP Chunks and Technical Terms , 2003, HLT-NAACL.

[10]  Claire Cardie,et al.  An Analysis of Statistical and Syntactic Phrases , 1997, RIAO.

[11]  Ellen M. Voorhees,et al.  The TREC-8 Question Answering Track Evaluation , 2000, TREC.

[12]  Noam Chomsky,et al.  वाक्यविन्यास का सैद्धान्तिक पक्ष = Aspects of the theory of syntax , 1965 .

[13]  Tomek Strzalkowski,et al.  Natural language information retrieval: progress report , 2000, Inf. Process. Manag..

[14]  Nina Wacholder,et al.  A model for quantitative evaluation of an end-to-end question-answering system , 2007, J. Assoc. Inf. Sci. Technol..

[15]  Gerard Salton,et al.  Another look at automatic text-retrieval systems , 1986, CACM.

[16]  Nicholas J. Belkin,et al.  Ask for Information Retrieval: Part I. Background and Theory , 1997, J. Documentation.

[17]  Judith A. Tessier Indexing: The state of our knowledge and the state of our ignorance , 1990 .

[18]  William R. Hersh,et al.  A Task-Oriented Approach to Retrieval System Evaluation. , 1995 .

[19]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[20]  Gerard Salton,et al.  Automatic indexing , 1980, ACM '80.

[21]  Nina Wacholder,et al.  User preference: A measure of query-term quality , 2006 .

[22]  Kenneth Ward Church,et al.  Termight: Identifying and Translating Technical Terminology , 1994, ANLP.

[23]  Cyril W. Cleverdon,et al.  The significance of the Cranfield tests on index languages , 1991, SIGIR '91.

[24]  J. Neter,et al.  Applied linear statistical models : regression, analysis of variance, and experimental designs , 1974 .

[25]  Louis M. Gomez,et al.  All the Right Words: Finding What You Want as a Function of Richness of Indexing Vocabulary. , 1990 .

[26]  David Ellis,et al.  The Physical and Cognitive Paradigms in Information Retrieval Research , 1992, J. Documentation.

[27]  Karen Spärck Jones Automatic language and information processing: rethinking evaluation , 2001, Natural Language Engineering.

[28]  E. Michael Keen,et al.  The Aberystwyth Index Languages Test. , 1973 .

[29]  Peter Willett,et al.  Readings in information retrieval , 1997 .

[30]  Susan T. Dumais,et al.  The vocabulary problem in human-system communication , 1987, CACM.

[31]  Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003 , 2003, HLT-NAACL.

[32]  Robert J. Sandusky Accessing and browsing information and communication , 2002, J. Assoc. Inf. Sci. Technol..

[33]  Louis M. Gomez,et al.  All the right words: Finding what you want as a function of richness of indexing vocabulary , 1990, J. Am. Soc. Inf. Sci..