Consensus-based evaluation framework for distributed information retrieval systems

Multi-agent systems have been attacking the challenges of information retrieval tasks on distributed environment. In this paper, we propose a consensus choice selection method based framework to evaluate the performance of cooperative information retrieval tasks of the multiple agents. Thereby, two well-known measurements, precision and recall, are extended to handle consensual closeness (i.e., local and global consensus) between the sets of retrieved results. We show that in a motivating example the proposed criteria are prone to solve the rigidity problem of classical precision and recall. More importantly, the retrieved results can be ranked with respect to the consensual score, and the ranking mechanism has been verified to be more reasonable.

[1]  M.P. Singh,et al.  Agents are everywhere , 1997, IEEE Internet Computing.

[2]  Schubert Foo,et al.  CQE: a collaborative querying environment , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[3]  Luigi Palopoli,et al.  A graph-based approach for extracting terminological properties from information sources with heterogeneous formats , 2004, Knowledge and Information Systems.

[4]  Shengli Wu,et al.  Distributed Information Retrieval: A Multi-Objective Resource Selection Approach , 2003, Int. J. Uncertain. Fuzziness Knowl. Based Syst..

[5]  Ellen M. Voorhees,et al.  The Collection Fusion Problem , 1994, TREC.

[6]  George Coulouris,et al.  Distributed systems - concepts and design , 1988 .

[7]  Alistair Moffat,et al.  Methodologies for distributed information retrieval , 1998, Proceedings. 18th International Conference on Distributed Computing Systems (Cat. No.98CB36183).

[8]  Hsin-Hsi Chen,et al.  Overview of CLIR Task at the Fourth NTCIR Workshop , 2004, NTCIR.

[9]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[10]  Pasquale Pagano,et al.  A Grid-Based Infrastructure for Distributed Retrieval , 2007, ECDL.

[11]  David Ellis.  The Dilemma of Measurement in Information Retrieval Research , 1996, J. Am. Soc. Inf. Sci..

[12]  Ngoc Thanh Nguyen,et al.  Consensus system for solving conflicts in distributed systems , 2002, Inf. Sci..

[13]  Ngoc Thanh Nguyen,et al.  A Consensus-Based Multi-agent Approach for Information Retrieval in Internet , 2006, International Conference on Computational Science.

[14]  James Nga-Kwok Liu,et al.  iJADE Web-miner: an intelligent agent framework for Internet shopping , 2004, IEEE Transactions on Knowledge and Data Engineering.

[15]  Jason J. Jung.  Ontological framework based on contextual mediation for collaborative information retrieval , 2007, Information Retrieval.

[16]  W. Bruce Croft,et al.  Searching distributed collections with inference networks , 1995, SIGIR '95.

[17]  William W. Cohen,et al.  NER Systems that Suit User’s Preferences: Adjusting the Recall-Precision Trade-off for Entity Extraction , 2006, NAACL.

[18]  Hsin-Hsi Chen,et al.  Overview of CLIR Task at the Third NTCIR Workshop , 2002, NTCIR.

[19]  Margaret-Anne D. Storey,et al.  Toward understanding the importance of gesture in distributed scientific collaboration , 2006, Knowledge and Information Systems.

[20]  Francisco Herrera,et al.  Incorporating filtering techniques in a fuzzy linguistic multi-agent model for information gathering on the web , 2004, Fuzzy Sets Syst..

[21]  Shengli Wu,et al.  Testing the cluster hypothesis in distributed information retrieval , 2006, Inf. Process. Manag..

[22]  Ralf Bierig,et al.  The DILIGENT framework for distributed information retrieval , 2007, SIGIR.

[23]  Michael D. Gordon,et al.  Recall-precision trade-off: A derivation , 1989, JASIS.

[24]  Elena V. Zudilova-Seinstra,et al.  On the role of individual human abilities in the design of adaptive user interfaces for scientific problem solving environments , 2007, Knowledge and Information Systems.

[25]  Thomas Krichel,et al.  Information retrieval performance measures for a current awareness report composition aid , 2007, Inf. Process. Manag..

[26]  Luis Gravano,et al.  Generalizing GlOSS to Vector-Space Databases and Broker Hierarchies , 1995, VLDB.

[27]  Tetsuya Sakai,et al.  On the reliability of information retrieval metrics based on graded relevance , 2007, Inf. Process. Manag..

[28]  Victor R. Lesser,et al.  Cooperative information-gathering: a distributed problem-solving approach , 1997, IEE Proc. Softw. Eng..

[29]  Jamie Callan,et al.  DISTRIBUTED INFORMATION RETRIEVAL , 2002 .

[30]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[31]  Fredric C. Gey,et al.  The relationship between recall and precision , 1994 .