Evaluating Multi-Model (Metadata-Semantic) Information Retrieval System

Abstract—In this paper, we present a variety of methodsto evaluate a multi-model information retrieval system. Theevaluation methods are divided into three steps: (1) evaluatinga metadata driven search engine, (2) evaluating a personalizedcluster-based semantic search engine, and (3) evaluating a dualrepresentation of the semantic user profile for personalized Websearch in an evolving domain. The first part of this paper presentsan overview of the most popular information retrieval models andtheir evaluation methods. The second part is dedicated to thespecific evaluation methods that we used to evaluate each modelof our platform. This platform is already being used by onlinestudents at WKU. As of 2006 the “HyperManyMedia” searchengine has been ranked number 24 on “The Ultimate Guide toUsing Open Courseware 1 ” (between Cambridge University andHarvard Business).Index Terms—Evaluation Methods, Precision, Recall, Informa-tion Retrieval I. I NTRODUCTION What is information? [7] divided information into three cate-gories: information-as-process, information-as-knowledge, andinformation-as-thing. It stated that “information is anythingthat can change person’s knowledge [7].”[16] looked at an information system from the user’sprospective. His framework is human centered, where “theinformation seeker defines the task, controls the interactionwith the search system, examines and extracts relevant in-formation, assesses the progress, and determines when theinformation-seeking is complete [16]”. The framework for thispaper is based on a student-centered approach, where a studentsearches for information, the system retrieves relevant infor-mation to the student’s query, the student uses the informationand, at the end, evaluates its relevance.[14] distinguished between Data and Wisdom. Data isequally available to any user, it is received, stored and retrieved

[1]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[2]  Michael K. Buckland,et al.  Information and information systems , 1991 .

[3]  G. Pasi,et al.  A Fuzzy Linguistic Approach Generalizing Boolean Information Retrieval: a Model and its Evaluation , 1993 .

[4]  Gary Marchionini,et al.  Information Seeking in Electronic Environments , 1995 .

[5]  Amanda Spink,et al.  An analysis of Web searching by European AlltheWeb.com users , 2005, Inf. Process. Manag..

[6]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[7]  George Tzanetakis,et al.  Audio Information Retrieval (AIR) Tools , 2000, ISMIR.

[8]  Bamshad Mobasher,et al.  Ontological User Profiles for Representing Context in Web Search , 2007, 2007 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Workshops.

[9]  Vijay V. Raghavan,et al.  On modeling of information retrieval concepts in vector spaces , 1987, TODS.

[10]  Michael I. Jordan,et al.  Modeling annotated data , 2003, SIGIR.

[11]  Daben Liu,et al.  Speech and language technologies for audio indexing and retrieval , 2000, Proceedings of the IEEE.

[12]  George Karypis,et al.  Soft clustering criterion functions for partitional document clustering: a summary of results , 2004, CIKM '04.

[13]  Olfa Nasraoui,et al.  Semantic Information Retrieval for Personalized E-Learning , 2008, 2008 20th IEEE International Conference on Tools with Artificial Intelligence.

[14]  William M. Pottenger.  D-HOTM: distributed higher order text mining , 2007, DG.O.

[15]  George Karypis,et al.  Evaluation of hierarchical clustering algorithms for document datasets , 2002, CIKM '02.

[16]  Olfa Nasraoui,et al.  Dual Representation of the Semantic User Profile for Personalized Web Search in an Evolving Domain , 2009, AAAI Spring Symposium: Social Semantic Web: Where Web 2.0 Meets Web 3.0.

[17]  Chris H. Q. Ding,et al.  Adaptive dimension reduction for clustering high dimensional data , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[18]  Charles H. Davis.  Information Storage and Retrieval. Robert R. Korfhage , 1999 .

[19]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[20]  Gerardo Canfora,et al.  Supporting change request assignment in open source development , 2006, SAC.

[21]  Miroslav Kubat,et al.  Search engine ranking efficiency evaluation tool , 2007, SGCS.

[22]  Sally Jo Cunningham,et al.  Usage analysis of a digital library , 1998, DL '98.

[23]  Christine L. Borgman,et al.  Social aspects of digital libraries (working session) , 1996, DL '96.

[24]  Stephen E. Robertson,et al.  A probabilistic model of information retrieval: development and comparative experiments - Part 1 , 2000, Inf. Process. Manag..

[25]  Malcolm Slaney,et al.  Mixtures of probability experts for audio retrieval and indexing , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.