On the Assessment of Expertise Profiles

Expertise retrieval has attracted significant interest in the field of information retrieval. Expert finding has been studied extensively, with less attention going to the complementary task of expert profiling, that is, automatically identifying topics about which a person is knowledgeable. We describe a test collection for expert profiling in which expert users have self-selected their knowledge areas. Motivated by the sparseness of this set of knowledge areas, we report on an assessment experiment in which academic experts judge a profile that has been automatically generated by state-of-the-art expert-profiling algorithms; optionally, experts can indicate a level of expertise for relevant areas. Experts may also give feedback on the quality of the systemgenerated knowledge areas. We report on a content analysis of these comments and gain insights into what aspects of profiles matter to experts. We provide an error analysis of the system-generated profiles, identifying factors that help explain why certain experts may be harder to profile than others. We also analyze the impact on evaluating expert-profiling systems of using selfselected versus judged system-generated knowledge areas as ground truth; they rank systems somewhat differently but detect about the same amount of pairwise significant differences despite the fact that the judged system-generated assessments are more sparse.

[1]  M. de Rijke,et al.  On the Evaluation of Entity Profiles , 2010, CLEF.

[2]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[3]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[4]  Falk Scholer,et al.  User performance versus precision measures for simple search tasks , 2006, SIGIR.

[5]  Donna K. Harman,et al.  Overview of the Second Text REtrieval Conference (TREC-2) , 1994, HLT.

[6]  David J. Sheskin,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 1997 .

[7]  Andrei Broder,et al.  A taxonomy of web search , 2002, SIGF.

[8]  C. J. van Rijsbergen,et al.  Report on the need for and provision of an 'ideal' information retrieval test collection , 1975 .

[9]  Peter Bailey,et al.  Overview of the TREC 2008 Enterprise Track , 2008, TREC.

[10]  Mark Sanderson,et al.  Information retrieval system evaluation: effort, sensitivity, and reliability , 2005, SIGIR '05.

[11]  Eugene Agichtein,et al.  Hits on question answer portals: exploration of link analysis for author ranking , 2007, SIGIR.

[12]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[13]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[14]  M. de Rijke,et al.  Determining Expert Profiles (With an Application to Expert Finding) , 2007, IJCAI.

[15]  Toine Bogers,et al.  Design and Evaluation of a University-Wide Expert Search Engine , 2009, ECIR.

[16]  M. de Rijke,et al.  A language modeling framework for expert finding , 2009, Inf. Process. Manag..

[17]  Ellen M. Voorhees,et al.  The effect of topic set size on retrieval experiment error , 2002, SIGIR '02.

[18]  B. L. Welch.  The generalisation of student's problems when several different population variances are involved. , 1947, Biometrika.

[19]  Filip Radlinski,et al.  Comparing the sensitivity of information retrieval metrics , 2010, SIGIR.

[20]  Gerard Salton,et al.  Automatic indexing , 1980, ACM '80.

[21]  M. de Rijke,et al.  Broad expertise retrieval in sparse data environments , 2007, SIGIR.

[22]  Filip Radlinski,et al.  How does clickthrough data reflect retrieval quality? , 2008, CIKM '08.

[23]  M. de Rijke,et al.  Expertise Retrieval , 2012, Found. Trends Inf. Retr..

[24]  Nick Craswell,et al.  Overview of the TREC 2005 Enterprise Track , 2005, TREC.

[25]  James Allan,et al.  A comparison of statistical significance tests for information retrieval evaluation , 2007, CIKM '07.

[26]  Peter Bailey,et al.  The CSIRO enterprise search test collection , 2007, SIGF.

[27]  Leif Azzopardi,et al.  Retrievability: an evaluation measure for higher order information access tasks , 2008, CIKM '08.

[28]  Katja Hofmann,et al.  A probabilistic method for inferring preferences from clicks , 2011, CIKM '11.

[29]  Mark Sanderson,et al.  Test Collection Based Evaluation of Information Retrieval Systems , 2010, Found. Trends Inf. Retr..

[30]  Welch Bl.  THE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED , 1947 .

[31]  S.J.J. Smith,et al.  Empirical Methods for Artificial Intelligence , 1995 .

[32]  Nick Craswell,et al.  Overview of the TREC 2006 Enterprise Track , 2006, TREC.

[33]  Donna K. Harman,et al.  Overview of the Fifth Text REtrieval Conference (TREC-5) , 1996, TREC.

[34]  Morten Hertzum,et al.  The information-seeking practices of engineers: searching for documents as well as for people , 2000, Inf. Process. Manag..

[35]  Peter Bailey,et al.  Overview of the TREC 2007 Enterprise Track , 2007, TREC.

[36]  Krisztian Balog,et al.  A User-Oriented Model for Expert Finding , 2011, ECIR.

[37]  Louise T. Su.  Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[38]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[39]  Mark Sanderson,et al.  Problems with Kendall's tau , 2007, SIGIR.

[40]  Ellen M. Voorhees,et al.  The fifth text REtrieval conference (TREC-5) , 1997 .

[41]  Ellen M. Voorhees.  Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[42]  Thorsten Joachims,et al.  Evaluating Retrieval Performance Using Clickthrough Data , 2003, Text Mining.

[43]  Harold Borko,et al.  Automatic indexing , 1981, ACM '81.

[44]  Jangwon Seo,et al.  Thread-based Expert Finding , 2009 .

[45]  Catherine L. Smith,et al.  User adaptation: good results from poor systems , 2008, SIGIR '08.

[46]  Maarten de Rijke,et al.  Contextual factors for finding similar experts , 2010, J. Assoc. Inf. Sci. Technol..

[47]  Donna K. Harman,et al.  Overview of the Ninth Text REtrieval Conference (TREC-9) , 2000, TREC.

[48]  James Allan,et al.  When will information retrieval be "good enough"? , 2005, SIGIR '05.

[49]  Donna K. Harman,et al.  Overview of the Reliable Information Access Workshop , 2009, Information Retrieval.