Combination of Multiple Retrieval Systems Using Rank-Score Function and Cognitive Diversity

Combining multiple retrieval systems is a commonly used method to improve the retrieval performance. However, it is still a challenging problem to figure out when and how the combined system can perform better than its individual systems. In this paper, we study these issues by using an information fusion paradigm: Combinatorial Fusion Analysis (CFA). TREC datasets are used as our experiment data. We measure the cognitive diversity between different individual systems by using a rank-score characteristic (RSC) function. Our results demonstrate that: 1) The performance of combination of p systems does not always increase with p, 2) Rank combination is better than score combination in particular when RSC diversity between two individual systems is large enough, and 3) combination of two systems can improve performance only if the two individual systems have relative good performance and are diverse.

[1]  Soon Myoung Chung,et al.  Combining Multiple Feature Selection Methods for Text Categorization by Using Rank-Score Characteristics , 2009, 2009 21st IEEE International Conference on Tools with Artificial Intelligence.

[2]  Christopher C. Vogt.  How much more is better? Characterising the effects of adding more IR Systems to a combination , 2000, RIAO.

[3]  D. Frank Hsu,et al.  Comparing Rank and Score Combination Methods for Data Fusion in Information Retrieval , 2005, Information Retrieval.

[4]  Ellen M. Voorhees,et al.  Overview of the seventh text retrieval conference (trec-7) [on-line] , 1999 .

[5]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  D. Frank Hsu,et al.  Combinatorial fusion with on-line learning algorithms , 2008, 2008 11th International Conference on Information Fusion.

[7]  D. Frank Hsu,et al.  Consensus Scoring Criteria for Improving Enrichment in Virtual Screening , 2005, J. Chem. Inf. Model..

[8]  Hong-wei Liu,et al.  The reconstruction of three-dimensional tree models from terrestrial LiDAR , 2011, 2011 IEEE International Conference on Computer Science and Automation Engineering.

[9]  D. Frank Hsu,et al.  Microarray Gene Expression Analysis Using Combinatorial Fusion , 2009, 2009 Ninth IEEE International Conference on Bioinformatics and BioEngineering.

[10]  Paul B. Kantor,et al.  Predicting the effectiveness of Naïve data fusion on the basis of system characteristics , 2000 .

[11]  D. Frank Hsu,et al.  Combining Multiple Retrieval Systems Using Combinatorial Fusion Analysis and Rank-Score Characteristic Function , 2011, 2011 14th IEEE International Conference on Computational Science and Engineering.

[12]  Kagan Tumer,et al.  Linear and Order Statistics Combiners for Pattern Classification , 1999, ArXiv.

[13]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[14]  Sergei Vassilvitskii,et al.  Generalized distances between rankings , 2010, WWW '10.

[15]  D. Frank Hsu,et al.  Combinatorial Fusion Analysis: Methods and Practices of Combining Multiple Scoring Systems , 2006 .

[16]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[17]  David Hawking,et al.  Overview of the TREC-2001 Web track , 2002 .

[18]  Donna K. Harman,et al.  Overview of the Eighth Text REtrieval Conference (TREC-8) , 1999, TREC.

[19]  Marc O. Ernst.  Decisions Made Better , 2010, Science.

[20]  Chuan Yi Tang,et al.  Feature Selection and Combination Criteria for Improving Accuracy in Protein Structure Prediction , 2007, IEEE Transactions on NanoBioscience.

[21]  Chuan Yi Tang,et al.  On the Diversity-Performance Relationship for Majority Voting in Classifier Ensembles , 2007, MCS.

[22]  Ludmila I. Kuncheva,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2004 .

[23]  Shaomeng Wang,et al.  How Does Consensus Scoring Work for Virtual Library Screening? An Idealized Computer Experiment , 2001, J. Chem. Inf. Comput. Sci..

[24]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[25]  Ellen M. Voorhees,et al.  Overview of the Seventh Text REtrieval Conference , 1998 .

[26]  Javed A. Aslam,et al.  Condorcet fusion for improved retrieval , 2002, CIKM '02.

[27]  D. F. Hsu,et al.  Combinatorial Fusion for Improving Portfolio Performance , 2010 .

[28]  D. Frank Hsu,et al.  Rank-Score Characteristics (RSC) Function and Cognitive Diversity , 2010, Brain Informatics.

[29]  Ofer Melnik,et al.  Mixed group ranks: preference and confidence in classifier combination , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Damian M. Lyons,et al.  Combining multiple scoring systems for target tracking using rank-score characteristics , 2009, Inf. Fusion.

[31]  P. Latham,et al.  References and Notes Supporting Online Material Materials and Methods Figs. S1 to S11 References Movie S1 Optimally Interacting Minds R�ports , 2022 .