An experiment in building profiles in information filtering: the role of context of user relevance feedback

An experiment was conducted to see how relevance feedback could be used to build and adjust profiles to improve the performance of filtering systems. Data was collected during the system interaction of 18 graduate students with SIFTER (Smart Information Filtering Technology for Electronic Resources), a filtering system that ranks incoming information based on users' profiles. The data set came from a collection of 6000 records concerning consumer health. In the first phase of the study, three different modes of profile acquisition were compared. The explicit mode allowed users to directly specify the profile; the implicit mode utilized relevance feedback to create and refine the profile; and the combined mode allowed users to initialize the profile and to continuously refine it using relevance feedback. Filtering performance, measured in terms of Normalized Precision, showed that the three approaches were significantly different (α = 0.05 and p = 0.012). The explicit mode of profile acquisition consistently produced superior results. Exclusive reliance on relevance feedback in the implicit mode resulted in inferior performance. The low performance obtained by the implicit acquisition mode motivated the second phase of the study, which aimed to clarify the role of context in relevance feedback judgments. An inductive content analysis of thinking aloud protocols showed dimensions that were highly situational, establishing the importance context plays in feedback relevance assessments. Results suggest the need for better representation of documents, profiles, and relevance feedback mechanisms that incorporate dimensions identified in this research.

[1]  Linda Schamber.  Users' criteria for evaluation in multimedia information seeking and use situations , 1991 .

[2]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[3]  Yoichi Shinoda,et al.  Information filtering based on user behavior analysis and best match text retrieval , 1994, SIGIR '94.

[4]  Sourav Bhattacharya,et al.  High-assurance systems , 1997, CACM.

[5]  Luz Marina Quiroga.  Personalized information organization : acquisition and modeling of users' interest profiles in information filtering systems , 2000 .

[6]  Tefko Saracevic,et al.  Modeling Interaction in Information Retrieval (IR): A Review and Proposal. , 1996 .

[7]  Gerard Salton,et al.  Improving Retrieval Performance by Relevance Feedback , 1997 .

[8]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[9]  Amanda Spink,et al.  Information Science: A Third Feedback Framework , 1997, J. Am. Soc. Inf. Sci..

[10]  Yoav Shoham,et al.  Fab: content-based, collaborative recommendation , 1997, CACM.

[11]  Yoav Shoham,et al.  Content-Based, Collaborative Recommendation. , 1997 .

[12]  Javed Mostafa,et al.  Filtering medical documents using automated and human classification methods , 1998 .

[13]  Judy Bateman.  Changes in Relevance Criteria: A Longitudinal Study. , 1998 .

[14]  John A. Self,et al.  Theoretical foundations for intelligent tutoring systems , 1990 .

[15]  Judy Kay,et al.  User Models for Customized Hypertext , 1997, Intelligent Hypertext.

[16]  Susan T. Dumais,et al.  Personalized information delivery: an analysis of information filtering methods , 1992, CACM.

[17]  Thomas W. Malone,et al.  How do people organize their desks?: Implications for the design of office information systems , 1983, TOIS.

[18]  G. Lakoff.  Women, fire, and dangerous things : what categories reveal about the mind , 1989 .

[19]  Nicholas J. Belkin,et al.  Information filtering and information retrieval: two sides of the same coin? , 1992, CACM.

[20]  Nicholas J. Belkin,et al.  Characteristics of Texts Affecting Relevance Judgments , 1993 .

[21]  Peter B. Danzig,et al.  Scalable Internet resource discovery: research problems and approaches , 1994, CACM.

[22]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[23]  Antonella De Angeli,et al.  A user-centered approach to user modeling , 1999 .

[24]  Javed Mostafa,et al.  Automatic classification using supervised learning in a medical document filtering application , 2000, Inf. Process. Manag..

[25]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[26]  Barbara Hanna Kwasnik.  The influence of context on classificatory behavior , 1989 .

[27]  Javed Mostafa,et al.  Empirical evaluation of explicit versus implicit acquisition of user profiles in information filtering systems , 1999, DL '99.

[28]  D. Case,et al.  The Collection and Use of Information by Some American Historians: A Study of Motives and Methods , 1991, The Library Quarterly.

[29]  Raya Fidel,et al.  Users' perception of the performance of a filtering system , 1997, SIGIR '97.

[30]  Peter B. Danzig,et al.  Scalable Internet resource discovery : research problems and approaches : Internet technology , 1994 .

[31]  Liliana Ardissono,et al.  Tailoring the interaction with users in electronic shops , 1999 .

[32]  G. Lakoff,et al.  Women, Fire, and Dangerous Things: What Categories Reveal about the Mind , 1988 .

[33]  Peretz Shoval,et al.  Stereotypes in Information Filtering Systems , 1997, Inf. Process. Manag..

[34]  Robert M. Losee,et al.  Feedback in Information Retrieval. , 1996 .

[35]  James M. Donovan.  Patron Expectations About Collocation , 1991 .

[36]  Douglas W. Oard,et al.  The State of the Art in Text Filtering , 1997, User Modeling and User-Adapted Interaction.

[37]  Deborah K. Barreau.  Context as a factor in personal information management systems , 1995 .