Effectiveness of Google keyword suggestion on users' relevance judgment: A mixed method approach to query expansion

Purpose – One of the most effective ways information retrieval (IR) systems including Web search engines can improve relevance performance is to provide their users with tools for facilitating query expansion. Search engines such as Google provide users with keyword suggest tools. This paper aims to investigate users’ criteria in relevance judgment regarding Google’s keywords suggest tool and to see how such keywords would lead to more relevant results from the viewpoint of users. Design/methodology/approach – Through a mixed method approach, quantitative and qualitative data were collected from 60 postgraduate students at Ferdowsi University of Mashhad, Iran, using four different instruments (questionnaire, thinking aloud technique, query logs and interviews). Findings – Among other criteria, the “relation between suggested keywords and the information need” (with the mean rate of 3.53 of four) was considered the most important by searchers in selecting suggested keywords for query expansion. Also, the “...

[1]  ChoiYoungok,et al.  User's relevance criteria in image retrieval in American history , 2002 .

[2]  Heting Chu,et al.  Factors affecting relevance judgment: a report from TREC Legal track , 2011, J. Documentation.

[3]  C. Lee Giles,et al.  Context and Page Analysis for Improved Web Search , 1998, IEEE Internet Comput..

[4]  Agissilaos Andreou,et al.  Ontologies and Query expansion , 2005 .

[5]  Susan Gauch,et al.  Intelligent Information Agents for the World Wide Web , 2003 .

[6]  Joemon M. Jose,et al.  How users assess Web pages for information seeking , 2005, J. Assoc. Inf. Sci. Technol..

[7]  T. Saracevic.  Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II , 2006 .

[8]  Yunjie Calvin Xu,et al.  User-Oriented Relevance Judgment: A Conceptual Model , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[9]  J. Pokorny,et al.  Web searching and information retrieval , 2004, Computing in Science & Engineering.

[10]  Efthimis N. Efthimiadis,et al.  Interactive query expansion: A user-based evaluation in a relevance feedback environment , 2000, J. Am. Soc. Inf. Sci..

[11]  Jarkko Kari,et al.  User-defined relevance criteria in web searching , 2006, J. Documentation.

[12]  Brian D. Davison,et al.  Explorations in tag suggestion and query expansion , 2008, SSM '08.

[13]  Ji-Rong Wen,et al.  Query clustering using user logs , 2002, TOIS.

[14]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[15]  Carol L. Barry.  User-Defined Relevance Criteria: An Exploratory Study , 1994, J. Am. Soc. Inf. Sci..

[16]  Bernard J. Jansen,et al.  The seventeen theoretical constructs of information searching and information retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[17]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[18]  John Yen,et al.  A fuzzy ontology-based abstract search engine and its user studies , 2001, 10th IEEE International Conference on Fuzzy Systems. (Cat. No.01CH37297).

[19]  Sandra G. Hirsh.  Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[20]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[21]  Peter G. Anick.  Using terminological feedback for web search refinement: a log-based study , 2003, SIGIR.

[22]  Fletcher T. H. Cole,et al.  An alternative approach to natural language query expansion in search engines: Text analysis of non-topical terms in Web documents , 2008, Inf. Process. Manag..

[23]  Linda Schamber,et al.  User Criteria in Relevance Evaluation: Toward Development of a Measurement Scale. , 1996 .

[24]  Xiangmin Zhang,et al.  Examination of relevance criteria choices and the information search process , 2009, J. Documentation.

[25]  W. Bruce Croft,et al.  Finding semantically similar questions based on their answers , 2005, SIGIR '05.

[26]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[27]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[28]  Paul Solomon,et al.  Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior , 1998, Inf. Process. Manag..

[29]  Amanda Spink,et al.  Multitasking information seeking and searching processes , 2002, J. Assoc. Inf. Sci. Technol..

[30]  Linda Schamber.  Users' criteria for evaluation in multimedia information seeking and use situations , 1991 .

[31]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[32]  G. G. Chowdhury,et al.  The internet and information retrieval research: a brief review , 1999, J. Documentation.

[33]  David Ellis,et al.  In search of the unknown user: indexing, hypertext and the world wide web , 1998, J. Documentation.

[34]  Masatoshi Yoshikawa,et al.  Adaptive web search based on user profile constructed without any effort from users , 2004, WWW '04.

[35]  Hsinchun Chen,et al.  Interactive term suggestion for users of digital libraries: using subject thesauri and co-occurrence lists for information retrieval , 1996, DL '96.

[36]  Ricardo A. Baeza-Yates,et al.  Query Recommendation Using Query Logs in Search Engines , 2004, EDBT Workshops.

[37]  Alia I. Abdelmoty,et al.  Ontology-Based Spatial Query Expansion in Information Retrieval , 2005, OTM Conferences.

[38]  Abdur Chowdhury,et al.  Automatic evaluation of world wide web search services , 2002, SIGIR '02.