We describe results from an application for relevance assessment in a setting related to multidocument summarization. For the task of characterizing given document collections by a short list of relevant terms, we have proposed the term utility function PxR. The measure is competitive to a variety of utility functions commonly used in text mining. Our function incorporates a user-definable parameter that allows for explicit, continuous trade-off between precision and recall, which was preferred by our users over the more opaque term utility functions from text mining. The F β measure is similar but not identical to our measure and will also be discussed. Despite our users' preference for a user-definable parameter, the improvement by setting different user-defined parameter values for each document collection are limited, and a static value for the parameter works almost as well. This seems to be true for the F β measure as well. A simple measure, SR, also performs competitively. In light of this evidence, a user-definable parameter seems to be unnecessary to achieve competitive performance.
[1]
Dunja Mladenic,et al.
Feature Subset Selection in Text-Learning
,
1998,
ECML.
[2]
C. J. van Rijsbergen,et al.
The selection of good search terms
,
1981,
Inf. Process. Manag..
[3]
C. J. van Rijsbergen,et al.
The use of hierarchic clustering in information retrieval
,
1971,
Inf. Storage Retr..
[4]
Eduard Hovy,et al.
Automated Text Summarization in SUMMARIST
,
1997,
ACL 1997.
[5]
Yiming Yang,et al.
A Comparative Study on Feature Selection in Text Categorization
,
1997,
ICML.
[6]
Johannes Fürnkranz,et al.
User profiling for the melvil knowledge retrieval system
,
2002,
Appl. Artif. Intell..
[7]
Yi-fang Brook Wu,et al.
Domain-specific keyphrase extraction
,
2005,
CIKM '05.
[8]
Chin-Yew Lin,et al.
Automated Text Summarization
,
2005,
IJCNLP.
[9]
Inderjeet Mani,et al.
Summarization Evaluation: An Overview
,
2001,
NTCIR.
[10]
Gordon W. Paynter,et al.
An Evaluation of Document Keyphrase Sets
,
2003,
J. Digit. Inf..