On the Evaluation of Entity Profiles

Entity profiling is the task of identifying and ranking descriptions of a given entity. The task may be viewed as one where the descriptions being sought are terms that need to be selected from a knowledge source (such as an ontology or thesaurus). In this case, entity profiling systems can be assessed by means of precision and recall values of the descriptive terms produced. However, recent evidence suggests thatmore sophisticated metrics are needed that go beyond mere lexical matching of system-produced descriptors against a ground truth, allowing for graded relevance and rewarding diversity in the list of descriptors returned. In this note, we motivate and propose such a metric.

[1]  Jimmy J. Lin,et al.  Will Pyramids Built of Nuggets Topple Over? , 2006, NAACL.

[2]  Terrence A. Brooks.  Topical subject Expertise and the Semantic Distance Model of Relevance Assessment , 1995, J. Documentation.

[3]  Fredric C. Gey,et al.  ENSM-SE at CLEF 2006 : Fuzzy Proximity Method with an Adhoc Influence Function in Evaluation of Multilingual and Multi-modal Information Retrieval 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Alicante, Spain , 2007 .

[4]  M. de Rijke,et al.  A language modeling framework for expert finding , 2009, Inf. Process. Manag..

[5]  Krisztian Balog,et al.  UvT Expert Collection documentation , 2007 .

[6]  M. de Rijke,et al.  Broad expertise retrieval in sparse data environments , 2007, SIGIR.

[7]  Valentin Jijkoun,et al.  Overview of the WiQA Task at CLEF 2006 , 2006, CLEF.

[8]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[9]  Charles L. A. Clarke,et al.  Novelty and diversity in information retrieval evaluation , 2008, SIGIR '08.

[10]  M. de Rijke,et al.  Determining Expert Profiles (With an Application to Expert Finding) , 2007, IJCAI.

[11]  Ellen M. Voorhees,et al.  Overview of the TREC 2004 Novelty Track. , 2005 .

[12]  M. de Rijke,et al.  Learning Semantic Query Suggestions , 2009, SEMWEB.

[13]  Mark Sanderson,et al.  The relationship between IR effectiveness measures and user satisfaction , 2007, SIGIR.

[14]  Jimmy J. Lin,et al.  Overview of the TREC 2007 Question Answering Track , 2008, TREC.

[15]  Abraham Bernstein,et al.  The Semantic Web - ISWC 2009, 8th International Semantic Web Conference, ISWC 2009, Chantilly, VA, USA, October 25-29, 2009. Proceedings , 2009, SEMWEB.

[16]  Katja Hofmann,et al.  The impact of document structure on keyphrase extraction , 2009, CIKM.

[17]  Hoa Trang Dang,et al.  Overview of the TREC 2006 Question Answering Track 99 , 2006, TREC.

[18]  Ellen M. Voorhees,et al.  The TREC 2005 robust track , 2006, SIGF.

[19]  Ellen M. Voorhees,et al.  Overview of the TREC 2002 Question Answering Track , 2003, TREC.