Using the information seeker to elicit construct models for search engine evaluation

With the rise of interactive search engines, direct access to information is available to the end user. Evaluation of these systems is clearly important for their comparison and development. From the perspective of the user engaged with the system in some information-seeking task, the range of relevant evaluation factors could be considerable. This article presents an investigation into the suitability of repertory grid technique eliciting a mental model of search engines in the user. Such a model comprises constructs held important to the user, which can be used to evaluate the system. The design of the repertory grid application is described and the data analyzed in two stages. The important issues in determining the suitability of the method are identified and the constructs are analyzed to determine their association, discriminatory ability, and clustering around a central overall rating. From this, it is concluded that repertory grid technique is appropriate for user-centered determination of evaluative constructs. Further analysis of the characteristics of the resulting construct set is undertaken towards establishing an underlying user model of the search engines, and further research is identified for its use in system evaluation.

[1]  Edie Rasmussen,et al.  Evaluating interactive systems in TREC , 1996 .

[2]  Beth Sandore.  Online Searching: What Measure Satisfaction?. , 1990 .

[3]  Brian R. Gaines,et al.  WebGrid: Knowledge elicitation and modeling on the Web , 1996, WebNet.

[4]  M. Gordon Hunter,et al.  The use of RepGrids to gather interview data about information systems analysts , 1997, Inf. Syst. J..

[5]  Mary Burke.  The Use of Repertory Grids To Develop a User-Driven Classification of a Collection of Digitized Photographs. , 2001 .

[6]  A. Stewart,et al.  Business applications of repertory grid , 1981 .

[7]  H. A. Fatmi,et al.  A novel definition of expert knowledge in expert systems , 1994, Proceedings of IEEE International Conference on Systems, Man and Cybernetics.

[8]  Stephen P. Harter.  Scientific inquiry: A model for online searching , 1984, J. Am. Soc. Inf. Sci..

[9]  Nicholas J. Belkin,et al.  Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval , 2001, Inf. Process. Manag..

[10]  Paul B. Kantor,et al.  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* , 1988 .

[11]  Nicholas J. Belkin,et al.  Cases, scripts, and information-seeking strategies: On the design of interactive information retrieval systems , 1995 .

[12]  Charles R. Hildreth,et al.  Accounting for users' inflated assessments of on-line catalogue search performance and usefulness: an experimental study , 2001, Inf. Res..

[13]  Hsinchun Chen,et al.  Cognitive process as a basis for intelligent retrieval systems design , 1991, Inf. Process. Manag..

[14]  Hong Xie,et al.  Supporting ease-of-use and user control: desired features and structure of Web-based online IR systems , 2003, Inf. Process. Manag..

[15]  Andrew Dillon,et al.  Towards a Classification of Text Types: A Repertory Grid Approach , 1990, Int. J. Man Mach. Stud..

[16]  R. N. Woolley.  On Becoming a Personal Scientis , 1982 .

[17]  Ping Zhang,et al.  Satisfiers and dissatisfiers: a two-factor model for website design and evaluation , 2000 .

[18]  David Ellis,et al.  A Behavioural Approach to Information Retrieval System Design , 1989, J. Documentation.

[19]  Mark Chignell,et al.  Assessment of the effects of user characteristics on mental models of information retrieval systems , 2001 .

[20]  Amanda Spink,et al.  Use of query reformulation and relevance feedback by Excite users , 2000, Internet Res..

[21]  Richard Bell,et al.  A manual for repertory grid technique , 1977 .

[22]  Robin Jeffries,et al.  Orienteering in an information landscape: how information seekers get from here to there , 1993, INTERCHI.

[23]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[24]  A. Bytheway,et al.  Factors affecting information systems’ success , 1996 .

[25]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[26]  G. Kelly.  The Psychology of Personal Constructs , 2020 .

[27]  Daryl J. D'Souza,et al.  Melbourne TREC-9 Experiments , 2000, TREC.

[28]  Soyeon Park.  Usability, user preferences, effectiveness, and user behaviors when searching individual and integrated full-text databases: implications for digital libraries , 2000 .

[29]  Marc Hassenzahl,et al.  Analysis of web sites with the repertory grid technique , 2001, CHI Extended Abstracts.

[30]  Peter Caputi,et al.  A COMPARISON OF TRIADIC AND DYADIC METHODS OF PERSONAL CONSTRUCT ELICITATION , 1999 .

[31]  Andrew Dillon,et al.  Designing Usable Electronic Text: Ergonomic Aspects Of Human Information Usage , 1994 .

[32]  Frances C. Johnson,et al.  Task dimensions of user evaluations of information retrieval systems , 2003, Inf. Res..

[33]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[34]  Tony Moynihan,et al.  An inventory of personal constructs for information systems project risk researchers , 1996, J. Inf. Technol..

[35]  Louise T. Su.  Value of Search Results as a Whole as the Best Single Measure of Information Retrieval Performance , 1998, Inf. Process. Manag..

[36]  Hsin-Liang Chen,et al.  Evaluation of Web-Based Search Engines from the End-User's Perspective: A Pilot Study , 1998 .

[37]  Keith Swigger,et al.  Validation of the repertory grid for use in modeling knowledge , 1992 .

[38]  Thong Ngee Goh,et al.  Service quality of Internet search engines , 1999, J. Inf. Sci..

[39]  Marcia J. Bates,et al.  Where should the person stop and the information search interface start? , 1990, Inf. Process. Manag..

[40]  Mildred L. G. Shaw,et al.  On Becoming a Personal Scientist: Interactive Computer Elicitation of Personal Models of the World , 1980 .

[41]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[42]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[43]  Cliff McKnight.  The personal construction of information space , 2000 .

[44]  Amanda Spink,et al.  From E-Sex to E-Commerce: Web Search Changes , 2002, Computer.

[45]  Hong Iris Xie.  Shifts of interactive intentions and information-seeking strategies in interactive information retrieval , 2000 .

[46]  Rachel Applegate,et al.  Models of User Satisfaction: Understanding False Positives , 1993 .