An empirical testing of user stereotypes of information retrieval systems

Stereotyping is a technique used in many information systems to represent user groups and/or to generate initial individual user models. However, there has been a lack of evidence on the accuracy of their use in representing users. We propose a formal evaluation method to test the accuracy or homogeneity of the stereotypes that are based on users' explicit characteristics. Using the method, the results of an empirical testing on 11 common user stereotypes of information retrieval (IR) systems are reported. The participants' memberships in the stereotypes were predicted using discriminant analysis, based on their IR knowledge. The actual membership and the predicted membership of each stereotype were compared. The data show that "librarians/IR professionals" is an accurate stereotype in representing its members, while some others, such as "undergraduate students" and "social sciences/humanities" users, are not accurate stereotypes. The data also demonstrate that based on the user's IR knowledge a stereotype can be made more accurate or homogeneous. The results show the promise that our method can help better detect the differences among stereotype members, and help with better stereotype design and user modeling. We assume that accurate stereotypes have better performance in user modeling and thus the system performance.Limitations and future directions of the study are discussed.

[1]  David Ellis,et al.  A Comparison of the Information seeking Patterns of researchers in the Physical and Social Sciences , 1993, J. Documentation.

[2]  Thomas Deelman,et al.  The development of behavior-based user models for a computer system , 1999 .

[3]  Richard Bell,et al.  A manual for repertory grid technique , 1977 .

[4]  David Dubin.  Measurement in information science , 1997 .

[5]  Keith Swigger,et al.  Validation of the repertory grid for use in modeling knowledge , 1992 .

[6]  Alfred Kobsa,et al.  KN-AHS: an adaptive hypertext client of the user modeling system BGP-MS , 1998 .

[7]  Peretz Shoval,et al.  Stereotypes in Information Filtering Systems , 1997, Inf. Process. Manag..

[8]  Johan Gustav Bellika,et al.  The virtual library secretary: A user model-based software agent , 2005, Personal Technologies.

[9]  Keith Swigger,et al.  Validation of the Repertory Grid for Use in Modeling Knowledge , 1992, J. Am. Soc. Inf. Sci..

[10]  C. Heckler.  Applied Discriminant Analysis , 1995 .

[11]  Mildred L. G. Shaw,et al.  On Becoming a Personal Scientist: Interactive Computer Elicitation of Personal Models of the World , 1980 .

[12]  Mildred L. G. Shaw,et al.  Validation in a Knowledge Support System: Construing and Consistency with Multiple Experts , 1988, Int. J. Man Mach. Stud..

[13]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[14]  S. Mulaik,et al.  Foundations of Factor Analysis , 1975 .

[15]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers , 1993, J. Am. Soc. Inf. Sci..

[16]  Debora Shaw.  Undergraduate Use of CD-ROM Databases: Observations of Human-Computer Interaction and Relevance Judgments , 1996 .

[17]  Peter Lund,et al.  Providing a networked future for interpersonal information retrieval: InfoVine and user modelling , 1998, Interact. Comput..

[18]  Baltasar Fernández-Manjón,et al.  Pragmatic user model implementation in an intelligent help system , 1998, Br. J. Educ. Technol..

[19]  DIMITRIOS PIERRAKOS,et al.  User Modeling and User-Adapted Interaction , 1994, User Modeling and User-Adapted Interaction.

[20]  Ellen M. Voorhees,et al.  Overview of the TREC 2002 Question Answering Track , 2003, TREC.

[21]  Fabio Paternò,et al.  Effective levels of adaptation to different types of users in interactive museum systems , 2000, J. Am. Soc. Inf. Sci..

[22]  Elaine Rich,et al.  Stereotypes and User Modeling , 1989 .

[23]  Ian H. Beaumont,et al.  User modelling in the interactive anatomy tutoring system ANATOM-TUTOR , 1994, User Modeling and User-Adapted Interaction.

[24]  Nipon Charoenkitkarn.  The effect of markup-querying on search pattern and performance in large-scale text retrieval , 1996 .

[25]  Tsvi Kuflik,et al.  Stereotype-based Versus Personal-based Filtering Rules in Information Filtering Systems , 2003, J. Assoc. Inf. Sci. Technol..

[26]  Charles H. Davis.  American Society for Information Science , 1984 .

[27]  Elaine Rich,et al.  User Modeling via Stereotypes , 1998, Cogn. Sci..

[28]  Bryce Allen,et al.  Information Tasks: Toward a User-Centered Approach to Information Systems , 1996 .

[29]  Robert B. Allen,et al.  User Models: Theory, Method, and Practice , 1990, Int. J. Man Mach. Stud..

[30]  Xiangmin Zhang.  Discriminant Analysis as a Machine Learning Method for Revision of User Stereotypes of Information Retrieval Systems , 2003 .

[31]  Christine L. Borgman,et al.  All users of information retrieval systems are not created equal: An exploration into individual differences , 1989, Inf. Process. Manag..

[32]  Louise T. Su.  A comprehensive and systematic model of user evaluation of Web search engines: II. An evaluation by undergraduates , 2003, J. Assoc. Inf. Sci. Technol..

[33]  D. Bannister,et al.  The evaluation of personal constructs , 1968 .

[34]  Max A. Woodbury,et al.  Individualizing user interfaces: application of the Grade of Membership (GoM) model for development of fuzzy user classes , 1994 .

[35]  E Collyer,et al.  On becoming a personal scientist , 1982 .

[36]  Josep Lluís de la Rosa i Esteva,et al.  A Taxonomy of Recommender Agents on the Internet , 2003, Artificial Intelligence Review.

[37]  Wei Ma.  A database selection expert system based on reference librarian's database selection strategy: A usability and empirical evaluation , 2002, J. Assoc. Inf. Sci. Technol..

[38]  Julita Vassileva,et al.  Adaptive Hypertext and Hypermedia , 1998, Springer Netherlands.

[39]  Mark H. Chignell,et al.  Assessment of the effects of user characteristics on mental models of information retrieval systems , 2001, J. Assoc. Inf. Sci. Technol..

[40]  Liwen Qiu.  Markov Models of Search State Patterns in a Hypertext Information Retrieval System , 1993, J. Am. Soc. Inf. Sci..

[41]  Alfred Kobsa,et al.  A Review and Analysis of Commercial User Modeling Servers for Personalization on the World Wide Web , 2000, User Modeling and User-Adapted Interaction.

[42]  Delia Neuman.  High school students' use of databases: results of a national Delphi study , 1995 .

[43]  Paul B. Kantor,et al.  A study of information seeking and retrieving. I. background and methodology , 1988 .

[44]  R. N. Woolley.  On Becoming a Personal Scientis , 1982 .

[45]  G. Kelly.  The Psychology of Personal Constructs , 2020 .

[46]  Peter Funk,et al.  Category-Based Filtering and User Stereotype Cases to Reduce the Latency Problem in Recommender Systems , 2002, ECCBR.

[47]  Peretz Shoval,et al.  Experimentation with an information filtering system that combines cognitive and sociological filtering integrated with user stereotypes , 1999, Decis. Support Syst..

[48]  David N. Chin.  KNOME: Modeling What the User Knows in UC , 1989 .

[49]  Javed Mostafa,et al.  Filtering Medical Documents Using Automated and Human Classification Methods , 1998, J. Am. Soc. Inf. Sci..

[50]  Alfred Kobsa,et al.  User Models in Dialog Systems , 1989, Symbolic Computation.

[51]  Giovanni Guida,et al.  User modeling in expert man-machine interfaces: a case study in intelligent information retrieval , 1990, IEEE Trans. Syst. Man Cybern..